Re: [ns] how to reduce processing cost of an awk script ... or adifferent problem?

[Lloyd Wood <l.wood@eim.surrey.ac.uk>]

My original script (available in the list archives etc.) has worked
fine for me on quite large flows (up to about a hundred thousand
packets on fairly hefty solaris hardware with lots of memory).

Since I wanted to examine and plot variation in packet delay, I stored
and updated all cumulative packet information in array elements per
packet id before sorting and outputting it at end. (Events at
intermediate nodes are not stored. The information relating to those
events updates the array[packet_id]). Note that the original is
intended for one, not multiple flows.

Since you only want average delay rather than per-packet delay
statistics, you could run a modified version of the awk script *for
each packet* based on flow_id and packet_id, outputting just delay
between endpoints or d if dropped to a file. Run the awk script in
loops that iterate through flow ids and packet ids; at the end you
will (via >> concatenation) have a single file containing end-to-end
delays or d's. Totalling these and computing average delay and number
of drops is pretty trivial; you've moved from being memory-intensive
to being disk-intensive.

Your problem is probably awk not being very efficient at manipulating
growing arrays. If you've got background traffic and your packet_ids
start with a large value, it may be worth subtracting an offset just
to get the array size down. I don't think awk handles sparse arrays
well.


On Wed, 13 Dec 2000,   wrote:

> Date: Wed, 13 Dec 2000 23:47:13 KST
> From:   <thesourmilk@hanmail.net>

Sticking to ascii chars would be nice.


> To: ns-users@ISI.EDU
> Subject: [ns] how to reduce processing cost of an awk script ... or a
>     different problem?
> 
> Hi,
> 
> I am trying to calculate total packet delay(ms) and drop rate(%) of 5 flows on a link using awk script...since the size of the tracefile is enormous, I have filtered it to print only lines with ($1 == d || $1 == r)..
> 
> My problem is that, despite the filtering, a memory allocation error occurs when I run awk script:
> awk -f NCmyawkscript out_scenario1.tr > result1
> awk: NCmyawkscript:12: (FILENAME=out_scenario1.tr FNR=890944) fatal: newnode: nextfree: can't allocate memory
> 
> As it is quite impossible to make the tracefile any shorter, I think I should modify my awk script (i.e. to reduce processing cost) but I can't figure out any other methods to compute delay and drop rate than the ones I have already used(shown below)..could anyone plz help me out on this? (it suddenly occurs to me that it might even be a totally different problem from memory shortage)
> 
> I would appreciate your help very much..thanks in advance!
> 
> 
> ******* NCmyawkscript *******
> BEGIN {
> # awk script to compute total packet loss (%) and average delay (ms)
> highest_packet_id = 0; total_duration = 0;
> num_drop = 0; num_rcvd = 0;
> }
> 
> {
> action = $1;
> time = $2;
> node_1 = $3;
> node_2 = $4;
> src = $5;
> flow_id = $8; 
> node_1_address = $9;
> node_2_address = $10; 
> seq_no = $11;
> packet_id = $12;
> 
> 
> if ( packet_id > highest_packet_id ) highest_packet_id = packet_id;
> 
> # assume you're not starting
> # traffic at 0.0.
> if ( start_time[packet_id] == 0 ) start_time[packet_id] = time;
> 
> # only useful for small unicast where packet_id doesn't wrap.
> # checking receive means avoiding recording drops
> if ( action != "d" ) {
> if ( action == "r" ) {
> # could test for receiving node_2_address or flow_id here.
> num_rcvd = num_rcvd + 1;
> end_time[packet_id] = time;
> }
> } else {
> num_drop = num_drop + 1;
> end_time[packet_id] = -1;
> }
> } 
> END {
> for ( packet_id = 0; packet_id <= highest_packet_id; packet_id++ ) {
> start = start_time[packet_id];
> end = end_time[packet_id];
> packet_duration = end - start;
> if ( start < end ) total_duration = total_duration + packet_duration;
> }
> printf("Packet Loss = %.6f \% \n", (num_drop/(highest_packet_id+1))*100);
> printf("Average Delay = %.6f ms \n", total_duration/num_rcvd); 
> }
> 
> ==================================================
> �츮 ���ͳ�, Daum
> ��� ���� ���� E-mail �ּ� �Ѹ��ϳ�
> ������ �ѱ� �˻����� Daum FIREBALL
> NO SPAM ķ����! : http://www.daum.net/event/nospam
> http://www.daum.net
> 
> 

<L.Wood@surrey.ac.uk>PGP<http://www.ee.surrey.ac.uk/Personal/L.Wood/>