Thoughts on 'performance' of ACS

[Zaid Abdulla]

On Mon, Sep 7, 2015, at 06:46 AM, Prashant Upadhyaya wrote:
> Hi guys,
> 
> I was wondering how to define the performance of an ACS in terms of
> its ability to handle all those CPE's out there.
> 
> Eg. should it be defined as the time taken to complete a typical
> Inform cycle during BOOT or BOOTSTRAP.
> Something like --
> TimeStart
> TCP connection initiation
> CPE Informs with BOOT
> No Firmware upgrade
> GPV
> SPV
> SPA
> .
> .
> TCP connection end
> TimeEnd
> 
> Clearly an ACS is 'fast' if the difference between the TimeStart and
> TimeEnd is minimum. Further, the time difference makes sense when 'N'
> CPE devices attack the ACS simultaneously. Then cumulatively, to
> services those N CPE's, the ACS will take a total time say T.
> Can we then say that the performance of ACS is defined by the
> parameter N/T which represents a throughput of servicing 'CPE's per
> second' by the ACS ?
> 
> Or am I clearly off the mark and there is a more meaningful way of
> defining the performance of an ACS, kindly do share with your
> experience.

A better metric to use is the number of TR-069 sessions completed in a
given period of time. The actual time a single session takes is
irrelevant because throughput is what matters. Ideally you want to test
that by spawning a number of virtual CPEs using the CPE simulator. A few
notes:

- Edit the simulator source code to specify the number of devices to
spawn and the inform interval.

- You can spawn a 1000 devices at 10 minute inform interval or a 100
devices at 1 minute inform interval. The results should be identical,
unless;

- you need to test for database memory bottleneck. For that you may want
to test using a DB server with small amount of memory and calculate your
actual memory requirements based on that.

- Not all sessions are equal. For better simulation, make sure your
presets are setup in the test environment similar to what you'd have in
production. Divide the times in your presets (e.g. parameter refresh
rate) by the same value you use to divide the inform interval. For
example, if you expect to have 100k devices at 100 minutes inform
interval, you can test server performance using 1000 CPE simulators at 1
minute inform interval. Make sure the times in your presets are divided
by 100 as well.

- The first time the CPE connects to the ACS, the ACS fetches all device
parameters and that's a heavy process. So make sure you do your measures
after all devices have been fully registered. You'll probably need to
spawn the simulators in batches (e.g. 100 at a time).

I have added these notes to the wiki
(https://github.com/zaidka/genieacs/wiki/Measuring-GenieACS-Performance).
Feel free to contribute to it. Let us know if you get any interesting
results :)

Zaid