Thoughts on 'performance' of ACS

[Prashant Upadhyaya]

Hi Zaid,

You mention 'CPE Simulator' -- do we already have one along with GenieACS.
If yes, can you please point me to that.

Regards
-Prashant


On Tue, Sep 8, 2015 at 8:45 PM, Zaid Abdulla <zaid at genieacs.com> wrote:
> On Mon, Sep 7, 2015, at 06:46 AM, Prashant Upadhyaya wrote:
>> Hi guys,
>>
>> I was wondering how to define the performance of an ACS in terms of
>> its ability to handle all those CPE's out there.
>>
>> Eg. should it be defined as the time taken to complete a typical
>> Inform cycle during BOOT or BOOTSTRAP.
>> Something like --
>> TimeStart
>> TCP connection initiation
>> CPE Informs with BOOT
>> No Firmware upgrade
>> GPV
>> SPV
>> SPA
>> .
>> .
>> TCP connection end
>> TimeEnd
>>
>> Clearly an ACS is 'fast' if the difference between the TimeStart and
>> TimeEnd is minimum. Further, the time difference makes sense when 'N'
>> CPE devices attack the ACS simultaneously. Then cumulatively, to
>> services those N CPE's, the ACS will take a total time say T.
>> Can we then say that the performance of ACS is defined by the
>> parameter N/T which represents a throughput of servicing 'CPE's per
>> second' by the ACS ?
>>
>> Or am I clearly off the mark and there is a more meaningful way of
>> defining the performance of an ACS, kindly do share with your
>> experience.
>
> A better metric to use is the number of TR-069 sessions completed in a
> given period of time. The actual time a single session takes is
> irrelevant because throughput is what matters. Ideally you want to test
> that by spawning a number of virtual CPEs using the CPE simulator. A few
> notes:
>
> - Edit the simulator source code to specify the number of devices to
> spawn and the inform interval.
>
> - You can spawn a 1000 devices at 10 minute inform interval or a 100
> devices at 1 minute inform interval. The results should be identical,
> unless;
>
> - you need to test for database memory bottleneck. For that you may want
> to test using a DB server with small amount of memory and calculate your
> actual memory requirements based on that.
>
> - Not all sessions are equal. For better simulation, make sure your
> presets are setup in the test environment similar to what you'd have in
> production. Divide the times in your presets (e.g. parameter refresh
> rate) by the same value you use to divide the inform interval. For
> example, if you expect to have 100k devices at 100 minutes inform
> interval, you can test server performance using 1000 CPE simulators at 1
> minute inform interval. Make sure the times in your presets are divided
> by 100 as well.
>
> - The first time the CPE connects to the ACS, the ACS fetches all device
> parameters and that's a heavy process. So make sure you do your measures
> after all devices have been fully registered. You'll probably need to
> spawn the simulators in batches (e.g. 100 at a time).
>
> I have added these notes to the wiki
> (https://github.com/zaidka/genieacs/wiki/Measuring-GenieACS-Performance).
> Feel free to contribute to it. Let us know if you get any interesting
> results :)
>
> Zaid
> _______________________________________________
> Users mailing list
> Users at lists.genieacs.com
> http://lists.genieacs.com/mailman/listinfo/users