From: info@mpauli.de
Date: Sat Nov 29 2003 - 15:51:42 GMT-3
Hi Jim,
could it be, that the prefixes will only be saved in memory once. Just as when they are reveived?
As long as you don't kill your BGB-session, you are not receiving (and not saving) your complete BGP-Table from your neighbor.
Cheers
Marcus
-------- Original Message --------
Subject: RE: Why Disabling BPG soft-reconfig did not reclaim Memory? (29-Nov-2003 2:32)
From: jim@powerpulse.cc
To: swm@emanon.com
> All,
>
> I had soft-reconfig on 2 of my production routers as well. I was taking 3
> full views (701, 6395, 7911) and passing 4 tables downstream ( 22584, 18687,
>
> 27496, 3718)
>
> I noticed that my memory changed VERY little as we suspected. Unfortunately,
>
> I didn't realize a sh ip bgp sum was the best data to capture.
>
> I did take a look at "sh mem sum" and "sh proc mem sort hold" ( BGP was
> obviously the tope candidate" I actually saw an increase in mem used.
>
> Fortunately, the GRP-B can take 512 MB and when I get some payola I will go
> to PRP-1 , and in the meantime bitch about how noisy some ASN's are but for
> me, either way it seems like the command really has no effect.
>
> The output are from routers that are identical. GSR12012, GRP-B, 256MB.
> 76911 comes into core02 and 701 and 6395 come into core 01. I have iBGP
> running between them.
>
> I guess more FYI than anything else.
>
> THANKS!
> Jim
>
> pwps-core01#sh mem summ
> Head Total(b) Used(b) Free(b) Lowest(b)
> Largest(b)
> Processor 5375D580 210381440 144389172 65992268 41647752
> 45936792
> Fast 5373D580 131080 129816 1264 1264
> 1212
>
>
> pwps-core02#sh mem summ
> Head Total(b) Used(b) Free(b) Lowest(b)
> Largest(b)
> Processor 5375D580 210381440 135321040 75060400 49995544
> 49298436
> Fast 5373D580 131080 129816 1264 1264
> 1212
>
>
> pwps-core01#sh proc mem sort hold
> Total: 210381440, Used: 144391112, Free: 65990328
> PID TTY Allocated Freed Holding Getbufs Retbufs Process
> 156 0 371707392 20959844 114648228 0 0 BGP Router
>
> pwps-core02#sh proc mem sorted hold
> Total: 210381440, Used: 135326460, Free: 75054980
> PID TTY Allocated Freed Holding Getbufs Retbufs Process
> 159 0 613600380 126390232 106952464 0 0 BGP Router
>
>
>
>
>
> AFTER THE RESETS!
>
> pwps-core01#sh mem summ
> Head Total(b) Used(b) Free(b) Lowest(b)
> Largest(b)
> Processor 5375D580 210381440 149174912 61206528 41647752
> 41580088
> Fast 5373D580 131080 129816 1264 1264
> 1212
>
>
> pwps-core02#sh mem summ
> Head Total(b) Used(b) Free(b) Lowest(b)
> Largest(b)
> Processor 5375D580 210381440 136519240 73862200 49995544
> 48314736
> Fast 5373D580 131080 129816 1264 1264
> 1212
>
>
> pwps-core01#sh proc mem sort hold
> Total: 210381440, Used: 149180644, Free: 61200796
> PID TTY Allocated Freed Holding Getbufs Retbufs Process
> 156 0 522355920 35517312 119280452 0 0 BGP Router
>
> pwps-core02#sh proc mem sort hold
> Total: 210381440, Used: 136524424, Free: 73857016
> PID TTY Allocated Freed Holding Getbufs Retbufs Process
> 159 0 635935864 134432128 108150468 0 0 BGP Router
>
>
> -----Original Message-----
> From: Scott Morris [mailto:swm@emanon.com]
> Sent: Friday, November 28, 2003 1:56 PM
> To: 'Anthony Pace'; 'Brian McGahan'; 'Jim Devane'; ccielab@groupstudy.com
> Cc: cchurch@wamnetgov.com; marcelo.medina@citigroup.com
> Subject: RE: Why Disabling BPG soft-reconfig did not reclaim Memory?
>
> It's my impression that the databases are set up in a particular fashion
> by the soft-reconfig command, regardless of whether you are or are not
> doing any particular filtering.
>
> How much memory is on your router(s)? Were you having any difficulties
> accepting all of the routes before?
>
> Short of going to a default route from your provider, you may consider a
> prefix-list idea (perhaps on their end if they will ponder it for you)
> to prevent routes of, say more specific than /20 from being sent to you,
> or being accepted on your end.
>
> I would be interested in the output of "show ip bgp summary" both with
> and without the soft-reconfig options... That'll tell a bit about the
> memory being used and how many entries et al. At least if you feel like
> resetting and/or rebooting things again! :)
>
> Scott
>
> -----Original Message-----
> From: nobody@groupstudy.com [mailto:nobody@groupstudy.com] On Behalf Of
> Anthony Pace
> Sent: Friday, November 28, 2003 2:21 PM
> To: Scott Morris; 'Brian McGahan'; 'Jim Devane'; ccielab@groupstudy.com
> Cc: cchurch@wamnetgov.com; marcelo.medina@citigroup.com
> Subject: Why Disabling BPG soft-reconfig did not reclaim Memory?
>
>
> I saw this thread and it was in vien of my situation. I had 2 production
> routers doing soft-reconfig and were at 90% memory utilization. They
> were taking in full tables from two providers(1 each). I removed
> soft-reconfig from both configs, cleared BGP and even rebooted routers
> but after they converged they were still using 90% memory.
>
> I now cannot view received routes, validating that the "untouched table
> is gone", but the memory utilization is identical.
>
> Is it possibel that instead of storing 2 tables, perhaps the router
> marks certain routes or indexes it some how to accomodate your filters?
> I was not really filtering it so maybe it never stored a second table?
>
> My next step is to only take in partial routes (my neigbor's neigbors)
>
> Tony Pace
>
>
> On Mon, 24 Nov 2003 16:17:11 -0500, "Scott Morris" <swm@emanon.com>
> said:
> > That's because the "policy" of being able to do a soft reset is
> > something that has to be set up in the database at the beginning of a
> > peering arrangement. So whenever you add it, the database needs to be
>
> > refreshed accordingly. Extra chunks of the database are used to be
> > able to do on-the-fly processing of the soft-reset.
> >
> > It's not so much that memory is held (the way we normally think of
> > applications), but more that the BGP database is allocated differently
>
> > in order to allow for better parsing, so that takes up memory. In my
> > experience, it hasn't necessarily been a 2:1 increase like someone
> > else had mentioned, but it may cause issues on otherwise over-laden
> > routers.
> >
> > Scott Morris, CCIE4 (R&S/ISP-Dial/Security/Service Provider) #4713,
> > CISSP, JNCIS, et al. IPExpert CCIE Program Manager
> > IPExpert Sr. Technical Instructor
> > swm@emanon.com/smorris@ipexpert.net
> > http://www.ipexpert.net
> >
> >
> > -----Original Message-----
> > From: nobody@groupstudy.com [mailto:nobody@groupstudy.com] On Behalf
> > Of Brian McGahan
> > Sent: Monday, November 24, 2003 3:52 PM
> > To: 'Jim Devane'; ccielab@groupstudy.com
> > Subject: RE: BGP Soft Reset (BGP RIB)
> >
> >
> > Jim,
> >
> > It appears that adding soft-reconfig forces a hard reset, but
> > removing it does not:
> >
> > R1(config)#router bgp 100
> > R1(config-router)#nei 204.12.1.254 remote-as 54
> > *Mar 4 20:15:07.836: %BGP-5-ADJCHANGE: neighbor 204.12.1.254 Up
> > R1(config-router)#nei 204.12.1.254 soft-reconfig in
> > *Mar 4 20:15:24.540: %BGP-5-ADJCHANGE: neighbor 204.12.1.254 Down
> Soft
> > reconfig change *Mar 4 20:16:02.326: %BGP-5-ADJCHANGE: neighbor
> > 204.12.1.254 Up
> > R1(config-router)#do sh ip bgp nei 204.12.1.254 received-routes
> <snip>
> > Total number of prefixes 10
> > R1(config-router)#no nei 204.12.1.254 soft-reconfig in
> > R1(config-router)#do sh ip bgp nei 204.12.1.254 received-routes %
> > Inbound soft reconfiguration not enabled on 204.12.1.254
> >
> >
> > It must need a hard clear to populate the 'received' routes in
> the
> > first place. I'd be interested to see your results of the memory
> > savings if any.
> >
> >
> > Brian McGahan, CCIE #8593
> > bmcgahan@internetworkexpert.com
> >
> > Internetwork Expert, Inc.
> > http://www.InternetworkExpert.com
> > Toll Free: 877-224-8987
> > Direct: 708-362-1418 (Outside the US and Canada)
> >
> >
> >
> > > -----Original Message-----
> > > From: Jim Devane [mailto:jim@powerpulse.cc]
> > > Sent: Monday, November 24, 2003 1:49 PM
> > > To: 'Brian McGahan'
> > > Subject: RE: BGP Soft Reset (BGP RIB)
> > >
> > >
> > > Brian,
> > >
> > > Actually, I don't believe the hard reset takes place automatically.
> > > At least not on a 12012 running 12.0.25T
> > >
> > > I went through my config's and found 2 neighbors that had the
> > > soft-reconfig in place and after I checked to see that the opposite
> > > site router was running 12.2 I removed it.. the BGP session has not
> > > had any change to it.
> > >
> > > Unless you meant something else and I am mis-understanding.
> > >
> > > I hope I can schedule a window this week to reset the session to see
>
> > > if there is any memory savings.
> > >
> > > Thanks,
> > > Jim
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: nobody@groupstudy.com [mailto:nobody@groupstudy.com] On Behalf
>
> > > Of Brian McGahan
> > > Sent: Monday, November 24, 2003 11:38 AM
> > > To: 'kurt'; ccielab@groupstudy.com
> > > Subject: RE: BGP Soft Reset (BGP RIB)
> > >
> > > Kurt,
> > >
> > > Correct, if route-refresh is supported you can 'clear
> > > ip bgp x.x.x.x in' without memory implications.
> > > Route-refresh is supported by default as of 12.0T.
> > >
> > > > also memory held by a previous soft (configured) reset will be
> > > > held
> > > > until a reboot ??
> > >
> > > This I'm not sure about, but I'm going to guess no. If you add
> or
> > > remove the 'soft-reconfiguration inbound' statement a hard clear of
> > > the BGP neighbor automatically takes place. Therefore any cached
> > > received routes will be automatically flushed.
> > >
> > >
> > > HTH,
> > >
> > > Brian McGahan, CCIE #8593
> > > bmcgahan@internetworkexpert.com
> > >
> > > Internetwork Expert, Inc.
> > > http://www.InternetworkExpert.com
> > > Toll Free: 877-224-8987
> > > Direct: 708-362-1418 (Outside the US and Canada)
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: kurt [mailto:kurt@cybernex.net]
> > > > Sent: Monday, November 24, 2003 1:18 PM
> > > > To: Brian McGahan
> > > > Subject: Re: BGP Soft Reset (BGP RIB)
> > > >
> > > >
> > > >
> > > > ok ,
> > > >
> > > > if i understand this correctly
> > > > i had been in the practice of setting
> > > > neighbor xxxxx soft-reconfiguration inbound
> > > > this causes the memory doubling issue
> > > > correct?
> > > >
> > > > if i do a
> > > > sh ip bgp neighbor
> > > > and it says route refresh is supported then i can
> > > > do a clear ip bgp * soft without memory implications right ??
> > > >
> > > > also memory held by a previous soft (configured) reset will be
> > > > held
> > > > until a reboot ??
> > > >
> > > > main-gw#sbn
> > > > BGP neighbor is , remote, external link
> > > > BGP version 4, remote router ID BGP state = Established,
> > > > up for 7w4d
> > > > Last read 00:00:00, hold time is 180, keepalive interval is 60
> > > > seconds
> > > > Neighbor capabilities:
> > > > Route refresh: advertised and received(old & new)
> > > > Address family IPv4 Unicast: advertised and received
> > > > Message statistics:
> > > > InQ depth is 7
> > > > OutQ depth is 0
> > > > Sent Rcvd
> > > > Opens: 2 2
> > > > Notifications: 1 0
> > > > Updates: 6 3448866
> > > > Keepalives: 148547 148540
> > > > Route Refresh: 1 0
> > > > Total: 148557 3597408
> > > > Default minimum time between advertisement runs is 30 seconds
> > > >
> > > > For address family: IPv4 Unicast
> > > > BGP table version 10566014, neighbor version 0
> > > > Index 1, Offset 0, Mask 0x2
> > > > Inbound path policy configured
> > > > Outgoing update prefix filter list is locally-originated
> > > >
> > > >
> > > > ----- Original Message -----
> > > > From: "Brian McGahan" <bmcgahan@internetworkexpert.com>
> > > > To: "'Kurt Kruegel'" <kurt@cybernex.net>;
> > > > <Ken.Farrington@barclayscapital.com>; <ccielab@groupstudy.com>
> > > > Sent: Monday, November 24, 2003 1:50 PM
> > > > Subject: RE: BGP Soft Reset (BGP RIB)
> > > >
> > > >
> > > > > Kurt,
> > > > >
> > > > > If route-refresh is supported then both 'clear ip bgp * in'
> > > > and 'clear
> > > > > ip bgp * soft in' will do route-refresh. If route-refresh is
> > > > > not
> > > > > supported and soft-reconfiguration is not configured you
> > > > will get the
> > > > > following error message:
> > > > >
> > > > > router#clear ip bgp 172.16.4.3 in
> > > > > %BGP: Inbound soft reconfig for 172.16.4.3 not possible as it
> > > > > has neither refresh capability, nor inbound soft reconfig
> > > > >
> > > > > router#clear ip bgp 172.16.4.3 soft in
> > > > > %BGP: Inbound soft reconfig for 172.16.4.3 not possible as it
> > > > > has neither refresh capability, nor inbound soft reconfig
> > > > >
> > > > >
> > > > >
> > > > > HTH,
> > > > >
> > > > > Brian McGahan, CCIE #8593 bmcgahan@internetworkexpert.com
> > > > >
> > > > > Internetwork Expert, Inc. http://www.InternetworkExpert.com Toll
>
> > > > > Free: 877-224-8987
> > > > > Direct: 708-362-1418 (Outside the US and Canada)
> > > > >
> > > > >
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Kurt Kruegel [mailto:kurt@cybernex.net]
> > > > > > Sent: Sunday, November 23, 2003 7:45 PM
> > > > > > To: Brian McGahan; Ken.Farrington@barclayscapital.com;
> > > > > > ccielab@groupstudy.com
> > > > > > Subject: Re: BGP Soft Reset (BGP RIB)
> > > > > >
> > > > > >
> > > > > > so ....
> > > > > > will
> > > > > > clear ip bgp in
> > > > > > force a route refresh ???
> > > > > >
> > > > > >
> > > > > >
> > > > > > ----- Original Message -----
> > > > > > From: "Brian McGahan" <bmcgahan@internetworkexpert.com>
> > > > > > To: <Ken.Farrington@barclayscapital.com>;
> > > <ccielab@groupstudy.com>
> > > > > > Sent: Sunday, November 23, 2003 8:21 PM
> > > > > > Subject: RE: BGP Soft Reset (BGP RIB)
> > > > > >
> > > > > >
> > > > > > > Ken,
> > > > > > >
> > > > > > > "Soft-reconfiguration" is Cisco's old proprietary way
> > > > of allowing
> > > > > > > changes in the BGP policy without tearing down the TCP
> > > > session and
> > > > > > > rebuilding it. "Route-refresh" is the standard and
> > > > > > preferred method
> > > > > > > to do so.
> > > > > > >
> > > > > > > Soft-reconfiguration must be preconfigured with the
> > > > > > 'neighbor x.x.x.x
> > > > > > > soft-reconfiguration inbound' bgp subcommand. With
> > > > > > soft-reconfig, the
> > > > > > > router caches prefixes received from a neighbor, then
> > > > installs a
> > > > > > > separate copy which is a result of configured inbound
> > > > > > > filters. The original copy can be seen by issuing the 'show
> > > > > > > ip bgp nei x.x.x.x received-routes' command. Using this
> > > > > > > method is not recommended, because the memory required on a
> > > > > > > per
> > > > neighbor basis
> > > > > > > can be
> > > > > > effectively
> > > > > > > doubled.
> > > > > > >
> > > > > > > Unlike soft-reconfig, route-refresh does not need to keep
> > > > > > two copies
> > > > > > > of the BGP table. There is no way to see the
> > > > unmodified received
> > > > > > > table other than going to the upstream side and doing a 'sh
> > > > > > ip bgp nei
> > > > > > > x.x.x.x advertised'. Route-refresh is part of the BGP
> > > > > > > capabilities exchange, and does not require any additional
> > > > > > > configuration.
> > > > > > >
> > > > > > >
> > > > > > > HTH,
> > > > > > >
> > > > > > > Brian McGahan, CCIE #8593 bmcgahan@internetworkexpert.com
> > > > > > >
> > > > > > > Internetwork Expert, Inc.
> > > http://www.InternetworkExpert.com Toll
> > > > > > > Free: 877-224-8987
> > > > > > > Direct: 708-362-1418 (Outside the US and Canada)
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: nobody@groupstudy.com
> > > [mailto:nobody@groupstudy.com] On
> > > > > > > > Behalf Of Bob Sinclair
> > > > > > > > Sent: Sunday, November 23, 2003 5:05 PM
> > > > > > > > To: Ken.Farrington@barclayscapital.com;
> > > ccielab@groupstudy.com
> > > > > > > > Subject: Re: BGP Soft Reset (BGP RIB)
> > > > > > > >
> > > > > > > >
> > > > > > > > Ken,
> > > > > > > >
> > > > > > > > The soft-reconfiguration inbound command is
> > > required if you
> > > > > > > > want to issue a
> > > > > > > > "neighbor received-routes" command. The received routes
> > > > > > > > command will show
> > > > > > > > you what the neighbor is sending you BEFORE any inbound
> > > > > > > > policies are applied. Very helpful!
> > > > > > > >
> > > > > > > > HTH,
> > > > > > > >
> > > > > > > > -Bob Sinclair
> > > > > > > > CCIE #10427, CISSP, MCSE bsinclair@netmasterclass.net
> > > > > > > >
> > > > > > > > ----- Original Message -----
> > > > > > > > From: <Ken.Farrington@barclayscapital.com>
> > > > > > > > To: <ccielab@groupstudy.com>
> > > > > > > > Sent: Sunday, November 23, 2003 5:05 PM
> > > > > > > > Subject: BGP Soft Reset (BGP RIB)
> > > > > > > >
> > > > > > > >
> > > > > > > > > Guys,
> > > > > > > > >
> > > > > > > > > Please can I confirm that nowadays you do not have
> > > > to use the "
> > > > > > > > > neighbor 200.200.1.1 soft-reconfiguration
> > > inbound" command.
> > > > > > > > >
> > > > > > > > > All my routers without this command have the
> > > > > > > > > route-refresh
> > > > > > > > capability
> > > > > > > > > on (please see output below), and as such and I can do a
> > > > > > > > >
> > > > > > > > > "cle ip bgp 200.200.1.1 soft in" or "cle ip bgp
> > > > > > > > 200.200.1.1 in"
> > > > > > > > >
> > > > > > > > > without configuring soft-reconfiguration under the
> > > > BGP process.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Also, I assume the router stores two copies of the
> > > > RIB, one
> > > > > > > > > unmodified
> > > > > > > > and
> > > > > > > > > one modified with inbound policies. If this is
> > > > > > correct, is there
> > > > > > > > > a
> > > > > > > > > way of showing the unmodified RIB ?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > TEST2#sh ip bgp nei 200.200.1.1
> > > > > > > > > BGP neighbor is 200.200.1.1, remote AS 6500,
> > > external link
> > > > > > > > > BGP version 4, remote router ID 155.195.16.161
> > > > > > > > > BGP state = Established, up for 00:06:48
> > > > > > > > > Last read 00:00:48, hold time is 180, keepalive
> > > > > > > > > interval
> > > > > > > > is 60 seconds
> > > > > > > > > Neighbor capabilities:
> > > > > > > > > Route refresh: advertised and received(new)
> > > > > > > > > <======================Please Note
> > > > > > > > > Address family IPv4 Unicast: advertised and received
> > > > > > > > > Received 1481 messages, 936 notifications, 0 in queue
> > > > > > > > > Sent 1537 messages, 0 notifications, 0 in queue
> > > > > > > > > Route refresh request: received 0, sent 6
> > > > > > > > > Default minimum time between advertisement runs
> > > > is 30 seconds
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > ------------------------------------------------------------------
> > > > --
> > > > > > > > --
> > > > > > > > > --
> > > > > > > > > For more information about Barclays Capital, please
> > > > > > > > > visit
> > > > > > > > > our web site at http://www.barcap.com.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Internet communications are not secure and therefore
> > > > > > the Barclays
> > > > > > > > > Group does not accept legal responsibility for the
> > > > > > contents of this
> > > > > > > > > message. Although the Barclays Group operates
> > > > > > > > > anti-virus
> > > > > > > > programmes,
> > > > > > > > > it does not accept responsibility for any damage
> > > > > > whatsoever that
> > > > > > > > > is
> > > > > > > > > caused by viruses being passed. Any views or opinions
> > > > > > > > presented are
> > > > > > > > > solely those of the author and do not necessarily
> > > represent
> > > > > > > > those of
> > > > > > > > > the Barclays Group. Replies to this email may be
> > > > > > monitored by the
> > > > > > > > > Barclays Group for operational or business reasons.
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > ------------------------------------------------------------------
> > > > --
> > > > > > > > --
> > > > > > > > > --
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > __________________________________________________________________
> > > > __
> > > > > > > > __
> > > > > > > > > _
> > > > > > > > > Please help support GroupStudy by purchasing your study
> > > > > > > > materials from:
> > > > > > > > > http://shop.groupstudy.com
> > > > > > > > >
> > > > > > > > > Subscription information may be found at:
> > > > > > > > > http://www.groupstudy.com/list/CCIELab.html
> > > > > > > >
> > > > > > > >
> > > ______________________________________________________________
> > > > > > > > _________
> > > > > > > > Please help support GroupStudy by purchasing your study
> > > > > > > > materials from: http://shop.groupstudy.com
> > > > > > > >
> > > > > > > > Subscription information may be found at:
> > > > > > > > http://www.groupstudy.com/list/CCIELab.html
> > > > > > >
> > > > > > >
> > > > > >
> > > >
> > > ____________________________________________________________________
> > > __
> > > > > > > _
> > > > > > > Please help support GroupStudy by purchasing your study
> > > > > > materials from:
> > > > > > > http://shop.groupstudy.com
> > > > > > >
> > > > > > > Subscription information may be found at:
> > > > > > > http://www.groupstudy.com/list/CCIELab.html
> > > > >
> > > > >
> > > > ______________________________________________________________
> > > > _________
> > > > > Please help support GroupStudy by purchasing your study
> > > > materials from:
> > > > > http://shop.groupstudy.com
> > > > >
> > > > > Subscription information may be found at:
> > > > > http://www.groupstudy.com/list/CCIELab.html
> > >
> > > ______________________________________________________________
> > > _________
> > > Please help support GroupStudy by purchasing your study
> > > materials from: http://shop.groupstudy.com
> > >
> > > Subscription information may be found at:
> > > http://www.groupstudy.com/list/CCIELab.html
> >
> > ______________________________________________________________________
> > _
> > Please help support GroupStudy by purchasing your study materials
> from:
> > http://shop.groupstudy.com
> >
> > Subscription information may be found at:
> > http://www.groupstudy.com/list/CCIELab.html
> >
> > ______________________________________________________________________
> > _
> > Please help support GroupStudy by purchasing your study materials
> from:
> > http://shop.groupstudy.com
> >
> > Subscription information may be found at:
> > http://www.groupstudy.com/list/CCIELab.html
> --
> Anthony Pace
> anthonypace@fastmail.fm
>
> --
> http://www.fastmail.fm - Same, same, but different
>
> _______________________________________________________________________
> Please help support GroupStudy by purchasing your study materials from:
> http://shop.groupstudy.com
>
> Subscription information may be found at:
> http://www.groupstudy.com/list/CCIELab.html
>
> _______________________________________________________________________
> Please help support GroupStudy by purchasing your study materials from:
> http://shop.groupstudy.com
>
> Subscription information may be found at:
> http://www.groupstudy.com/list/CCIELab.html
To: jim@powerpulse.cc
anthonypace@fastmail.fm
bmcgahan@internetworkexpert.com
ccielab@groupstudy.com
Cc: cchurch@wamnetgov.com
marcelo.medina@citigroup.com
This archive was generated by hypermail 2.1.4 : Fri Dec 12 2003 - 12:29:19 GMT-3