filter enhancements

[phaworth]

While we are pondering on filter it would be nice to consider arrays:

Code: Select all

filter [keys|elements of] <container> [ not ] matching [ pattern ] <pattern>

Great idea!
Pete

[phaworth]

Folks,
Sorry to be thick but I'm still not getting sure why we need the "items" keyword.

Let's say tVar contains:

abc,123,def
abc,12345,xyz

filter items of tVar with "123" returns line 1 and filter items of tVar with "123*" returns both lines, right? But filter tVar with "*123" and filter tVar with "*123*" achieves the same result, doesn't it?

Maybe there's other use cases I'm not thinking of but "items" seems like it's just a convenience rather than serving a useful purpose.

Hopefully someone will enlighten me!

Pete

[Janschenkel]

FYI: I'vs submitted a pull request for the fix that filter should honour the lineDelimiter property.

Cheers,

Jan Schenkel.

[Janschenkel]

Support for filtering 'items' aims for consistency with the 'sort' command; looking at the 'sort' command I'm slightly confused about its output.
I always though that sort items meant that each line would be sorted separately, but that doesn't appear to be the case; in fact, I got strange results when sorting items in a multiline list.
Maybe we should discuss here how we expect 'filter items' to work. And leave 'sort items' to a separate but related discussion.

Base principle: 'item' chunk type < 'line' chunk type

Assuming we have a variable with the following data:

Code: Select all

foo,bar
baz,qux

And the following command:

Code: Select all

filter items of theList with "b*"

My interpretation would be that the output is:

Code: Select all

bar
baz

In other words, filter handles each line separately, and filters the items within.

Does that make sense?

Jan Schenkel.

[phaworth]

Ah OK, I see where my confusion was - it's not the filtering end of things, it's the output format. I've never used the sort command in conjunction with items but it may cause confusion if sort items works differently than filter items.

So, if the list being filtered was:

Code: Select all

foo,bar,bad
baz,qux

...would you expect the output from your filter to be:

Code: Select all

bar
bad
baz

...or

Code: Select all

bar,bad
baz

Pete

[Janschenkel]

@phaworth
My aim is to have consistent meaning, so I would indeed expect:

Code: Select all

bar,bad
baz

(thus preserving the line structure)

Having said that, it seems that 'sort items' doesn't work that way at the moment.
It merely swaps out the lineDelimiter for the itemDelimiter, ignoring the concept of item being 'smaller than' a line in chunk terms.

Maybe someone at the run rev team can clarify the intention?
Then at least we can implement a 'filter items' command consistent with 'sort items' semantics.

Jan Schenkel.

[phaworth]

Just tried "sort items" on:

Code: Select all

foo,bar,bad
baz,qut

The result was:

Code: Select all

bad
baz,bar,foo,qut

I can't figure out what the intent of that result is. Maybe "sort items" is broken?

Pete

[Janschenkel]

@phaworth
Yah, that's what threw me off when I did a similar test: 'sort items' appears to merely use the itemDelimiter instead of the lineDelimiter and promptly ignore the chunk hierarchy.
I don't think that's what it was meant to do, but changing it now would be a behavioural change which RunRev isn't too eager to introduce.
So I won't code the 'filter items' option until we get some clarification..

Jan Schenkel.

[BvG]

sort items ignores lines. just as sort lines ignores words. If you want to sort stuff that contains lines by item, it's your job to make sure that you only sort within those lines, not the engines. You specified items, and that's what you get

Similar, if you filter by items, it's not the engine's job to care wether you're using lines or not. So sorting a multi-line item list will be confusing (and only usefull in very rare corner cases), but sorting a list of items will work just fine.

At least that's what I'd expect.

[phaworth]

sort items ignores lines. just as sort lines ignores words. If you want to sort stuff that contains lines by item, it's your job to make sure that you only sort within those lines, not the engines. You specified items, and that's what you get

Similar, if you filter by items, it's not the engine's job to care wether you're using lines or not. So sorting a multi-line item list will be confusing (and only usefull in very rare corner cases), but sorting a list of items will work just fine.

At least that's what I'd expect.

OK, I understand now, thanks Bjornke. I redid the test with just one line in the variable and ti worked as expected with the items in the correct sorted order.

So I guess filter would work by returning just the items that qualified. So filtering "foo,bar,bad" with "b*" would return "bad,bar". Nice.

Pete

[monte]

Hmm... I don't think I've ever used sort items on a multi-line container. My guess is the intention of sort items is for single lines containers. For multi-line containers we probably want:

Code: Select all

filter lines of theList with "b*" by chunkExpression of each

[phaworth]

Hmm... I don't think I've ever used sort items on a multi-line container. My guess is the intention of sort items is for single lines containers. For multi-line containers we probably want:

Code: Select all

filter lines of theList with "b*" by chunkExpression of each

Yep, that's pretty much it, but I think that's destined for phase II.

[Janschenkel]

Once we get around to the FIlterMapReduce phase, you could script something like:

Code: Select all

filter theList by item 2 of each begins with "b"
filter lines of theList by MyFilterFunction(each)

But that's food for a separate discussion - I first wanted to get some smaller items out of the way

Jan Schenkel.

[monte]

Ah... ok, serves me right for skim reading

[mwieder]

I've just gotten back into town now, and I'm a bit jetlagged in addition to coming into the discussion late.
In general I like the way this is going: fix the filter syntax first, then start making the syntax consistent across other commands, then expand on the syntax.

I *would*, however, like to take to extrapolation to array syntax into a separate thread and separate pull requests, because I think there's a lot of work to do on arrays and a lot of pitfalls along the way. Dealing with arrays of a single dimension looks easy enough by expanding the current syntax, but gets very sticky when going to multiple dimensions. And I'd like to avoid going down a path now that would have to be reverted.