counting occurrences

[adventuresofgreg]

Hi: What do you think would be the fastest and most efficient method of counting the number of occurrences of a number in a list? ie: (1023.45,1023.65,1023.45.... etc). I need to do this with each number in the list, and the list could be thousands of numbers. A logical method would be to stick them into an array, then just start comparing and incrementing a counter for each number, but it seems that there should be an easier way.

thanks.

Greg K

[Dixie]

Your friend here would be the filter command...

Dixie

[bn]

Hi Greg,

I think an array would be a good idea.
make a new stack, a button and a field:

Code: Select all

on mouseUp
   # generate data, 50000 random floating point numbers 
   put 0.378 into tSource
   repeat 50000
      put  random(10000) + tSource & "," after tCollect
   end repeat
   delete last char of tCollect -- a comma
   # end generate data
   
   # count occurences
   lock screen
   put the milliseconds into tStart
   put "" into tArray
   repeat for each item anItem in tCollect
      add 1 to tArray[anItem]
   end repeat
   combine tArray by return and tab
   set the itemDelimiter to tab
   sort numeric tArray by item 2 of each -- sort ascending by occurence, sort by item 1 if you want to sort by number
   put tArray into field 1
   set the tabStops of field 1 to 70 -- make it a bit prettier
   
   unlock screen
   
   put the milliseconds - tStart  -- the time to count occurences in milliseconds in the message box
end mouseUp

it is very fast and easy to code.

Kind regards
Bernd

[dunbarx]

Hi.

Use an array to count for you. If your number was in a variable "myNumber" and your comma delimited list was in "myList" then:

Code: Select all

on mouseUp
   repeat for each item tItem in myList
      add 1 to tNumberCount[tItem]
   end repeat
   answer tNumberCount[myNumber]
end mouseup

[adventuresofgreg]

Thank you! That helped a lot. I am playing around with a way to adjust the 'sensitivity' of the match count. So, I would like a "3" to match a "3.25" if my sensitivity value was .25. One way I thought of doing this would be to duplicate each number in the list, and add & subtract this "sensitivity" value to each duplicate. Is my logic right?

Thanks
Greg

[dunbarx]

Hi.

So much fun.

If you have your numbers in the lines of a field named "listOfItems", then using your ".25" as a variance and "3" as a "base", put this in a button script:

Code: Select all

on mouseUp
   put fld "listOfItems" into mylist
   put 3 into targetInteger
   put ".25"  into variance
      repeat for each line tLine in myList
            if abs(tLine - targetInteger) <= variance then  add 1 to tNumberCount[trunc(tLine + variance)]
      end repeat
      answer tNumberCount[targetInteger]
end mouseup

I filled my list with numbers, some all around the targetInteger, like 3, 3.1, 3.5, 2.9, 2.2,4000000, etc. The inclusive sum of all the numbers between 2.75 and 3.25 are totaled, since the "base" number was "3". Can you see what each line does, and why?

Craig Newman

[bn]

As Craig said: a lot of fun.

I tried with div and mod. This assumes you want to have the decimal part in .5 steps. I.e. what is equal or larger x.75 and what is equal and smaller x.25 will be a whole number, the rest will be x.5
I furthermore assumes that there are only 2 decimals. Feed this a return delimited list of your testnumbers.


Edit: this is the corrected version of the code, it replaces a prior version. I forgot to increment tDiv if tMod > .74

Code: Select all

on mouseUp
   put field 1 into tData
   repeat for each line aLine in tData
      put aLine div 1 into tDiv
      put aLine mod 1 into tMod
      if tMod < 0.26 or tMod > 0.74 then
   if tMod > 0.74 then add 1 to tDiv
         put aLine & tab & tDiv & return after tCollect
      else
         put aLine & tab & tDiv & .5 & return after tCollect
      end if
   end repeat
   delete last char of tCollect
   put tCollect into field 2
end mouseUp

To generate testnumbers I used

Code: Select all

on mouseUp
   put 0.10 into tSeed
   put 0.15 into tInc
   repeat with i = 1 to 200
      add tInc to tSeed
      put tSeed & return after tCollect
   end repeat
   delete last char of tCollect
   put tCollect into field 1
end mouseUp

You could use this to preprocess your numbers and feed the array with the preprocessed numbers. That way you get at your occurences right away.
Kind regards
Bernd

[adventuresofgreg]

Yes - I see what you are doing, and where I was going wrong by expanding my number list to include the sensitivity value. My list of numbers are actually prices for a futures contract, and those prices fluctuate by fixed tick amounts (ie: by .25, so prices would look like: 1001.00, 1001.25, 1000.75, 1000.50, etc).

So, if my 'sensitivity value' for determining match counts was ".25", then in the above case, the first number 1001.00 should return a MATCH for 1001.00, 1001.25, and 1000.75.

If my sensitivity value was 2 * the tick, so 2* .25, then I would need to REPEAT the above process twice, the first time adding and subtracting .25 to each number, and the second time adding and subtracting .5 to each number - and then counting the occurances.

Where I was going wrong, was I was basically doing the process ONCE, and adding and subtracting (2*.25) to each number - which is wrong because it leaves out (1*.25).

So this is what I ended up with - seems to work. And I think it is similar to your example "TV" is the sensitivity value, and "mult" is the number of times we need to add and subtract that value to the base number:

# count occurences
set the itemdelimiter to ","
put "" into HighCounts

put 0 into n
repeat for Mult+1
repeat for each item anItem in HIGHgroup
add 1 to HighCounts[anItem+(TV*n)]
add 1 to HighCounts[anItem-(TV*n)]
end repeat
put n+1 into n
end repeat

Thanks for your help! I'm going to post a YouTube video to show you guys what I'm making.

[adventuresofgreg]

Here is a YouTube video (no sound) of my app in action. It's still very much a work in progress. The counting occurrences handler can be seen when the small control panel with the 2 sliders on it appears along with some multi-colored horizontal lines. The slider changes the lookback period (ie: move slider to the right, and the handler gets more historical data to count duplicates, and left, fewer. The slider on the bottom adjusts the sensitivity, with "0" (shown) being an exact match, and 1,2,3, etc being that number x's the minimum tick (this results in higher match counts where prices are close, but not exact).

The Buy / Sell control panel that pops out is an API that transmits orders to my broker.

Make sure you change the QUALITY to 1080 HD or you won't be able to see the details (recorded on a large monitor).

http://youtu.be/HX4apnYHRZk

http://youtu.be/HX4apnYHRZk

[dunbarx]

I am missing something here.

If your range was 2 * the tick, why bother with 1 * the tick? In other words, if you are using "<=", you encompass both. So in my script, set the variance to 0.5 and you also get the 0.25 swings. And it does not matter if the variance is plus or minus.

I am not getting this, I suppose.

Craig Newman

[adventuresofgreg]

Craig - yes, you are correct. I realized this after reading your code again.

[glenn9]

Hi Greg,

I think an array would be a good idea.
make a new stack, a button and a field:

Code: Select all

on mouseUp
   # generate data, 50000 random floating point numbers 
   put 0.378 into tSource
   repeat 50000
      put  random(10000) + tSource & "," after tCollect
   end repeat
   delete last char of tCollect -- a comma
   # end generate data
   
   # count occurences
   lock screen
   put the milliseconds into tStart
   put "" into tArray
   repeat for each item anItem in tCollect
      add 1 to tArray[anItem]
   end repeat
   combine tArray by return and tab
   set the itemDelimiter to tab
   sort numeric tArray by item 2 of each -- sort ascending by occurence, sort by item 1 if you want to sort by number
   put tArray into field 1
   set the tabStops of field 1 to 70 -- make it a bit prettier
   
   unlock screen
   
   put the milliseconds - tStart  -- the time to count occurences in milliseconds in the message box
end mouseUp

it is very fast and easy to code.

Kind regards
Bernd

Dear Bernd,

Many thanks for this post. I'm a newbie to LC and it has solved a problem re counting occurences that I've been trying to do for weeks!

Although your code works beautifully, and I can apply it to my problem (counting occurences of words in a list), I'm unable to understand the logic of why the loop works. Grateful if you could explain the logic behind the loop that enables the occurences of items to be counted! Many thanks, Kind regards, Glenn

Code: Select all

    repeat for each item anItem in tCollect
      add 1 to tArray[anItem]
   end repeat

[bn]

Dear Bernd,

Many thanks for this post. I'm a newbie to LC and it has solved a problem re counting occurences that I've been trying to do for weeks!

Although your code works beautifully, and I can apply it to my problem (counting occurences of words in a list), I'm unable to understand the logic of why the loop works. Grateful if you could explain the logic behind the loop that enables the occurences of items to be counted! Many thanks, Kind regards, Glenn

Code: Select all

    repeat for each item anItem in tCollect
      add 1 to tArray[anItem]
   end repeat

Dear Glenn,

The code is a bit cryptic because it implicitly creates the array entry.

the repeat for each form of the repeat loop takes in this case (for each item) the current item into the variable anItem.

consider:

Code: Select all

put "abc,def,ghi" into tMyItems
repeat for each item anItem in tMyItems

this will iterate over the items in tMyItems 3 times
anItem will in turn contain "abc" then "def" and then "ghi"

in the next line of code

Code: Select all

add 1 to tArray[anItem]

anItem becomes a key of the array tArray. This key = anItem is augmented by 1 each time that this specific item occurs in variable tMyItems.

tArray will look like this

Code: Select all

tArray
          abc
                1
          def
               1
          ghi
               1

(look at this in the debugger it is is not clear from the description)

Then to get the number of occurences you query the array for the specific keys.

the "repeat for each loop" also works for words/lines/chars etc.
It is a very fast form, a lot faster then "repeat with i = 1 to xyz because Livecode does have to go back and count again.
But you loose the reference to the item number which you get from "i" in "repeat with i = 1 to xyz".

I hope this explains it a bit.

Kind regards
Bernd

[glenn9]

Hi Bernd,

Many thanks indeed for the explanation, I now understand!

(I still find it amazing that just 2 lines of code sorts out this issue of counting occurences!)

Kind regards,

Glenn

[MichaelBluejay]

I know this is an ancient thread, but people will still find it when searching, so here's a pretty simple way to count occurrences.

The trick is that itemDelimiter can be more than one character.

Code: Select all

put "a million balloons were allowed" into myString
set the itemdelimiter to "ll" -- (two lowercase L's)
answer the number of items of myString -- returns 4

"answer item 2 of myString" would return "ion ba".