Mike's Solaris Blog

Jumpstarting large machines

nospam@example.com (Mike Scott) — Sat, 24 Apr 2010 06:08:25 -0400

I recently found myself with an interesting problem to solve while jumpstarting some very large SPARC machines. Aside from some of the neat functionality that has been stuck in there since I last looked at it in detail, there's absolutely nothing new about jumpstarting machines, and although I've jumpstarted dozens/hundreds of machines of varying sizes over the years, there was significant expectation (and risk) to upgrading these particular machines and I wanted to make sure that I got it right first time, every time.

These particular machines are 32-way SPARC64 V based Fujitsu machines (actually 64-way, but each chassis is divided into two partitions). They have numerious I/O controllers (off the top of my head, I'd estimate about 16 controllers), with an even split of both traditional SCSI and fibrechannel controllers that were installed at various points during the lifecycle of the old OS. Most controllers have devices visible on them - either locally installed hard drives, or LUNs visible across the SAN.

My plan was to retain the pair of drives from the old Operating System (Solaris 8), and install the new OS onto a fresh pair of drives which were already connected - this would facilitate an easy back-out, should it be required. I could not guarantee that the controller numbers would not renumber during the re-install - and to have Jumpstart accidentally select the wrong devices could lead to the old OS image being overwritten, data loss, or simply just the wrong devices being selected so that the disks are not mirrored between separate system boards as intended.

Now, armed with the engineering manual of the PrimePOWER server in question, I could have researched the probe order of the PCI cards, and correleated this with the existing installation, but to be frank, I'm way too lazy to do that. I'd rather find another less error-prone method of forcing Jumpstart to select the correct devices for install. Additionally, given that a full boot/POST/reboot cycle on these machines can take in excess of an hour, I was keen to ensure that the outage window for the work was kept to an absolute mininum - this meant that I had to be absolutely sure that I was going to get it right first time.

I think I found a good, simple solution. In fact, I'm surprised that this functionality isn't already allowed in the jumpstart profile.

Within Solaris, we already have a unique device identifier that will persist across reboots and OS re-installs - the physical device path. So, rather than specificying the intended devices using the traditional cXtYdZ notation, we should be able to specify a physical device path (for example on this particular machine type it would take the format something like this: /pci@86,4000/scsi@4/sd@0,0). In fact - we don't need to specify the entire physical path - the only requirement is that we specify enough to uniquely identify one device, but in most cases it's probably safer to fully specify that path.

In order to facilitiate this, I used a jumpstart derived profile, and using a very basic "begin" script that will check the system hostname, and according to a case statement will identify which physical devices are appropriate for this host. It will then lookup the logical access path in cXtYdZ format and generate the necessary profile for it.

I'll not fully reproduce the script here because it contains references to client config and other functionality that I may write about in the future, but the following code fragments ought to give you the idea (hostnames have been changed to protect the innocent):

case `hostname` in



pw2500a)

        SOL10_ROOT=/pci@86,4000/scsi@4/sd@0,0

        SOL10_MIRROR=/pci@a4,4000/scsi@4/sd@0,0

        ;;



pw2500b)

        SOL10_ROOT=/pci@8a,4000/scsi@4/sd@0,0

        SOL10_MIRROR=/pci@94,4000/scsi@4/sd@0,0

        ;; 

*)

        gameOver "ERROR: `hostname` not found in config"

        ;;

esac

######################

# Translate a physical device to a logical cXtXdX 

translateDevice() {

        if [ $# -ne 1 ] ; then

                gameOver "ERROR : translateDevice() called with no args"

        fi

        DEVICE=$(ls -l /dev/dsk/c*s0 |grep $1 |awk '{print $(NF-2)}') 

        if [ -z "$DEVICE" ] ; then

                gameOver "ERROR: $1 not found in /dev/dsk/"

        fi

        if [ $(echo $DEVICE |wc -w) -gt 1 ] ; then

                gameOver "ERROR: $1 found, but returned multiple disks ($DEVICE)"

        fi

        print $DEVICE | cut -d/ -f4 | sed 's/s0$//'

}

######################

# Main program

if [ -z "$SOL10_ROOT" ] ; then

        gameOver "ERROR: SOL10_ROOT not defined for `hostname`"

fi 

SOL10_ROOT=$(translateDevice $SOL10_ROOT)

if [ -z "$SOL10_ROOT" ] ; then

        exit 1

fi

SOL10_ROOT="${SOL10_ROOT}sXSECTIONX"

SOL10_DEVS=$SOL10_ROOT

if [ ! -z "$SOL10_MIRROR" ] ; then

        SOL10_MIRROR=$(translateDevice $SOL10_MIRROR)

        if [ -z "$SOL10_MIRROR" ] ; then

                exit 1

        fi

        SOL10_MIRROR="${SOL10_MIRROR}sXSECTIONX"

        SOL10_DEVS="$SOL10_ROOT $SOL10_MIRROR"

fi

print "filesys mirror $SOL10_DEVS 10240 /"              |sed 's/XSECTIONX/0/g'

print "filesys mirror $SOL10_DEVS 4096 swap"            |sed 's/XSECTIONX/1/g'

print "filesys mirror $SOL10_DEVS 6144 /var"            |sed 's/XSECTIONX/3/g'

print "filesys mirror $SOL10_DEVS free /export"         |sed 's/XSECTIONX/5/g'

print "metadb $SOL10_ROOT size 8192 count 3"            |sed 's/XSECTIONX/7/g'

if [ ! -z $SOL10_MIRROR ] ; then

print "metadb $SOL10_MIRROR size 8192 count 3"          |sed 's/XSECTIONX/7/g'

fi

(any errors or omissions in that script are likely to be due to me copy/pasting it into this entry..).

Importantly, this method also allows the user to run this script on the
host prior to the jumpstart actually being performed, and it should
still accurately identify the correct devices using their current
logical access paths. A useful pre-install check:

$ ./getDiskLayout

filesys mirror c1t0d0s0 c5t0d0s0 10240 /

filesys mirror c1t0d0s1 c5t0d0s1 4096 swap

filesys mirror c1t0d0s3 c5t0d0s3 6144 /var

filesys mirror c1t0d0s5 c5t0d0s5 free /export

metadb c1t0d0s7 size 8192 count 3

metadb c5t0d0s7 size 8192 count 3

$

The end result - every single machine built first time correctly, without a significant amount of manual investigation into the physical device path and PCI probe order. When using derived profiles, there are many ways that the same functionality could be implemented, and this illustrates just one of those methods.

Shared memory controls in Solaris 10

nospam@example.com (Mike Scott) — Tue, 28 Aug 2007 03:06:06 -0400

The controls for IPC settings such as shared memory and semaphores have changes somewhat in Solaris 10. This seems to be a cause of regular confusion, and recently a friend asked me to clarify how they ought to be configured.

The new facilities in Solaris 10 makes such settings dynamic, but the controls to prevent overallocation are still there, albeit in a slightly modified form.

There are two approaches these days..

/etc/system (yawn, requires a reboot - why would you?)
Projects

The IPC settings are entirely controlled by projects these days.. the ability to set values in /etc/system is there solely for backwards compatibility, and simply sets the default values for the projects, and therefore should be considered deprecated.

Probably the best way to explain the use of projects to set default values is to run through a quick worked example of setting the shared memory max allocation (shmmax), with comments inline:

$ uname -a

SunOS deepthought 5.11 snv_69 i86pc i386 i86pc

It's reporting 5.11 as it's an OpenSolaris build, but same priciples apply to 5.10

$ id -a

uid=500(mike) gid=100(users) groups=100(users)

The user we're running as has no specific privs, such as would be the case with an Oracle application account (for example). There is no settings in /etc/system, so we're running with default values.

As an aside, this is running under VMware server on a Linux host with 256MB memory allocated to the VM. The default value of shmmax is calculated as physmem/4, so we ought to see maximum allocation around about 64MB

We can check what the max shm is going to be using Solaris 10 prctl command:

$ prctl -n project.max-shm-memory $$

process: 6905: ksh -o vi

NAME    PRIVILEGE       VALUE    FLAG   ACTION                       RECIPIENT

project.max-shm-memory

         privileged      61.9MB      -   deny                                 -

         system          16.0EB    max   deny                                 -

$

That's reporting 61.9MB, which is pretty close to the expected 64MB. I wonder where the other 3.1MB went?

And to prove the point, I've just written a wee program that tests this, by creating (and destroying) increasingly large segments of shared memory (incrememnting by 1MB at each iteration). It stops when it fails and reports the last created seg.

$ ./findmax

Shared seg created with 61.0 MB

shmget failed!

$

So, here's how we increase it.. We need root privs, or a friendly sysadmin for this, as we're adding a new project for this user only:

$ su

Password:

# projadd -U mike -K "project.max-shm-memory=(priv,80MB,deny)" user.mike

#

Back on as me now.. (notice I'm logging on again by su'ing back to myself, rather than just exiting the "su" shell - this is deliberate)

# su - mike

Sun Microsystems Inc.   SunOS 5.11      snv_69  October 2007

$ prctl -n project.max-shm-memory $$

process: 7044: ksh

NAME    PRIVILEGE       VALUE    FLAG   ACTION                       RECIPIENT

project.max-shm-memory

         privileged      80.0MB      -   deny                                 -

         system          16.0EB    max   deny                                 -

$

$ ./findmax

Shared seg created with 80.0 MB

shmget failed!

$

'crypt'ing in Linux..

nospam@example.com (Mike Scott) — Mon, 07 Aug 2006 11:21:13 -0400

Anybody else noticed that the Solaris 'crypt' command isn't in common use in the Linux community?

This proved to be an issue for me a little while back while transferring some data between companies and systems. Fortunately a quick visit to OpenSolaris provided the source, which only required a small modification in order to allow it to be compiled for use under Linux..

67c67
< ret = des_crypt(buf, &buf[8]);
---
> ret = crypt(buf, &buf[8]);

Yet another advantage to Solaris being open-sourced...

Demystifiying ACp..

nospam@example.com (Mike Scott) — Tue, 25 Jul 2006 09:16:45 -0400

After a bit of healthy debate in the office about the merits and
implications of using adaptive copy mode (ACp) with EMC's SRDF, I wanted to clarify my own
thoughts on how it operates, and the benefits of using it.

Firstly, I guess, for the uninitiated, it's a question of what is it exactly is it? - Well, EMC has their SRDF (Symmetrix Remote Data Facility) data replication protocol that allows data to be replicated over distance from one array to another.

Normally, whereever possible, when running replication at the hardware layer like this (where the hardware has no concept of application transactional consistency), you need the replication to perform synchronously with the I/Os. This essentially means that when a write operation is performed on the source array, it is flushed from disk and committed to the destination (i.e. remote) array before the operation is ack'd to the server as being completed.

Clearly there is a time overhead with this kind of arrangement, dictated by two factors - the (relatively constant, dependant on array loading) "rippling" effect of the write operation having to pass through two arrays before getting a final "commit", and also the transmission and acknowledgement time (as dictated by the speed of light and the distance between the arrays, which can be many kilometres apart).

From this (the aggregate of two times the transmission latency and the write latency for the remote array), it can be easily seen that the distance has an immediate bottleneck (not necessisarily in bandwidth, but certainly in transaction time - roundtrip times of 20-30ms is not unusual).

As the long-haul links between the arrays become congested, performance will rapidly degrade, and because of the synchronous nature your application will run slowly (I've seen write operations in excess of 300-400ms reported by the OS in bad cases)

In contrast, an async transfer mode will commit the write operation to local storage while transmitting the write to the remote array in the backgroup. This gives performance comparable to that of a non-SRDF setup, at the expense of the risk of missing I/O transactions in the event of a link or primary site failure. Because of the asynchronous nature of the SRDF transaction, even when the long-haul links between the two arrays are congested, your application will perform well and the SRDF updates will conclude at the next available opportunity.

And so, back to ACp - a compromise between the two.

The trouble with Synchronous is that you don't necessarily want your application slowing down at every busy spot during the day, perhaps you want to make more efficient usage of your bandwidth by playing the odds. You might, for example feel that running your long haul link at an extremely high utilisation is more cost effective than an upgrade in bandwidth. ACp will allow you to do this, but with the caveat that during peak load, you will be slightly out of sync.

ACp introduces the concept of the skew value.. essentially a threshold, counted in number of write operations. Below the skew value, the device pairing operates in async mode, and above the threshold it switches to synchonous mode (the skew value normally defaults to 65536 operations).

So, for example, playing the odds.. If, by running your long haul link at a higher utilisation meant that you couldn't keep your local app running quickly enough due to the link contention and latency, ACp may help. You might not be fully consistent one hundred percent of the day, but 98% might be good enough, especially when coupled with the local recovery abilities of journalled filesystems and modern DBMS.

Personally, I have only ever used ACp to mitigate high-bandwidth situations, for example bringing a new pair of arrays online, and having to perform a full SRDF establish of many terabytes of data. However.. the gotcha here is, depending on what else is using your long-haul links.. you may want to set your other services to ACp also, lest your high-bandwidth transfers have a knock-on effect on other (synchronous) pairings that are going on over the same links.

Back in Blighty..

nospam@example.com (Mike Scott) — Tue, 25 Jul 2006 07:18:01 -0400

I've been debating whether or not to post this here, as it's not strictly a unixy thing, but what the hell.. I'm ethused enough to be boring enough people locally about it, so a few more isn't going to harm the world.

I'm just back in the country after spending two and a half weeks touring Eastern Europe by motorbike. It's the latest in a series of trips that I've made over the years.

It wasn't the longest, the fastest, or probably not even the most challenging (I think the Sahara desert two years ago wins that prize), but it was certainly the most entertaining. Fifteen countries and 4,135miles in 18 days. Brilliant.

Full story here

What do you do with your decommissioned hardware?

nospam@example.com (Mike Scott) — Wed, 21 Jun 2006 05:09:07 -0400

We recently purchased a few drives to fit into one of our servers. I'll not say who supplied the drives, or even what kind of server, suffice to say it's not Sun, and it runs Solaris.

What arrived was a bit disturbing.

The drives were purchased as new parts, and were delivered fully sealed and packaged as a new drive.

We installed the drives, and noticed that the VTOC on one of the drives was a bit different to what we'd normally expect, so we checked with fstyp, and found (to our surprise) a few UFS filesystems.

To our astonishment we then proceeded to mount up a root filesystem from a system that was clearly from another UK company (in particular, a large mobile telephone provider).

The root filesystem in question appeared to have been shut down cleanly, and there was no tell-tale signs to suggest that it was failed drive that had been remanufactured. Presumably it is from a system that has been traded in, or otherwise decommissioned.

Out of interest, we noted that the "root" and "oracle" accounts had encryptions in /etc/shadow, and proceeded to run a password cracker against it.

So - what do YOU do with your decommissioned kit?, and perhaps more pertinently, what does your vendor do with your decommissioned kit?

Expect the unexpected

nospam@example.com (Mike Scott) — Sat, 20 May 2006 08:08:27 -0400

Working a particularly troublesome problem this week reminded me of two things: I realised (not for the first time in my career) that sometimes the unlikliest root cause will be there to bite you in the ass.

It also reminded me of a phrase once used by the late John Peel:

I never make stupid mistakes. Only very, very clever ones.

You see, I'm currently working a plan to decommission a whole raft of legacy E10000 systems from the current client.

We moved a four-board domain last week from one datacentre to another, using a previously idle domain on the destination platform. Everything went smoothly - all three SANs were visible, the network appeared to come up, we re-applied a new Veritas license (because the hostid had changed). However - after I did a final reboot before leaving for the night I noted that I was having trouble connecting to the public network interface. In fact, I was seeing in excess of 95% packet loss on inbound traffic.

Backing out the change was not a (pleasant) option, so instead I updated DNS and the defaultrouter to divert all traffic through the backup network as a workaround, and left for the night (it was getting very late, and this wasn't a production host).

The following day I rustled up a member of the comms team, and we spent an afternoon in the datacentre, trying to troubleshoot in a structured manner. The results of this testing was annoyingly inconclusive - the problem was showing up on multiple cables, multiple NICs, multiple hosts and multiple switch ports.

As a workaround to try and ensure that the public and backup traffic don't beat each other up, we decided to connect the public network interface into the backup subnet.

That was the very, very clever mistake. Normally this wouldn't be a problem, because normally we have local-mac-address set, to ensure that each NIC on the machine has a unique Ethernet address. Of course - this caused problems at the first point that significant traffic was being passed through the backup interface - but when the problems were reported I noted the configuration on "ifconfig", it struck me that the behaviour that we were seeing was remarkably similar to the other problem with the public network.

I quickly logged onto a machine with a connection to that public network, performed a broadcast ping ("ping -s 255.255.255.255"), then checked the ARP table and compared it with the Ethernet address of the problem domain. My suspiscions were confirmed... we have two E10k domains with the same Ethernet address (and hostid), and both are set with "local-mac-address" to false. This must have been a deliberate action, as Sun must have generated the necessary key to allow the domain to be generated like so, but the real reason why this sysadmin trap was configured like so has been lost over the years.

Sheesh...

Vx Storage Foundation Free!

nospam@example.com (Mike Scott) — Thu, 18 May 2006 14:15:53 -0400

Good grief - I've only just heard about this
There are of course restrictions (Linux/Solaris x64 and only a limted number of volumes). Is this evidence of the Veritas, cough Symantec coming under ZFS pressure?

Failover article online

nospam@example.com (Mike Scott) — Thu, 18 May 2006 13:59:09 -0400

As requested, I've put the Clones, Failovers and Migrations article online here. As always - feedback is appreciated, in fact.. consider it mandatory if you find it useful

Rsync saves the day

nospam@example.com (Mike Scott) — Sat, 04 Mar 2006 12:39:25 -0500

Installing a replacement drive and being unable to shrink a filesystem to fit to the smaller replacement drive. I encountered an interesting problem today - how to copy a large dataset containing many hardlinked files.

Hard links are used relatively rarely, and so don't normally cause an issue - however - there was an interesting solution using rsync..

At home I have a Red Hat, Fedora based server, which earns its' keep by serving a number of roles. The most fundamental facility that it provides is fileserver, with a pair of (mirrored) 160GB drives.

Earlier in the week one of these drives failed on me, which after finding that its' warranty had expired, I proceeded to order a replacement.

With the benefit of hindsight, I guess I ought to have ordered an obviously larger drive (say, 200GB). Without that piece of common sense.. drive replacements are always a bit of a lottery - especially when the previous make/model is no longer a viable option. Needless to say - the replacement turned out to be about 4GB smaller than the original.

Being a hardened Solaris geek, I guess I'm just a little too used to having the benefit of VxFS in production environments, which would have made shrinking the filesystem an absolute doddle.

However, with the aid of Google, it began to be clear that shrinking the existing ext3 filesystem so that I wouldn't truncate it on the smaller disk was going to be difficult.

"No problem" thinks I, proceeding to dump/restore the filesystem across onto the new disk. Shortly after, it dawned on me that this wasn't going to work and that I had a problem.

You see, amongst other things, this server acts as the backup server for my laptop using "BackupPC". BackupPC is quite simply an astounding piece of software, and suits my purpose absolutely perfectly.

The problem lies in the way that BackupPC stores its' backups on the filesystem - as it backs up files from clients, it MD5s the file and indexes the checksums, thereby identifying whether that file has been previously backed up.

If the file has been backed up already, it knows from the MD5 hashed index that it doesn't need to make another copy, and so just creates a hard link to the original.

And therein lies the problem...

In the backup pool, there can be files with literally hundreds of hardlinks, which, dump/restore (even tar) doesn't handle very well at all.. Rather than a 1MB file, with a hundred links, there would be one hundred 1MB files after I had finished copying.

It took me a while, but I spotted a previously unnoticed option to rsync:

-H, --hard-links preserve hard links

Ideal - kudos once again to the rsync developers..

Filesystem copied, my backups are preserved and the metadevice re-setup. Back in business.

Solaris 10 in production.. finally.

nospam@example.com (Mike Scott) — Mon, 13 Feb 2006 18:40:39 -0500

Something strange happened here today. I'm not sure I get some of the reasons of why it happened.. partly Client company politics, I guess.

I've been pushing the use of Solaris 10 quite heavily recently, and after trying to drum up enthusaism for a while, found an ideal project to make the first production machines.

This was all well and good, and the techies that I'd spoken to on the subject were keen.

However - it seemed that this move worried some management, and it took a meeting to drill down to what the concerns were.

Solaris 10 has had a lot of good publicity since its' launch.. we all know it's got some great things in it (and some great things to come). However - with such huge advances over Solaris 9 - it's gotta be a whole heap different from the previous release, and therefore difficult to manage the transition, right?

In truth - from an operational perspective, there isn't much difference in a Solaris 10 build from its' predecessor.

Yes, there are a whole bunch of interesting features out there to try out, but the vast majority of them are optional - take for example, containers - it's a piece that I find intriguing, and lined with opportunities, but it's still developing... I'm not yet convinced that I would advocate their use on an important production system.

Dtrace - a great troubleshooting tool, which is going to be desparately useful, it's there if you want it. You can ignore it, and the system will run just fine.

There is however, one exception.. the SMF. This is the biggest change to the operational interface, and it's going to be the one that trips up those coming to Solaris 10 for the first time.

Goodness knows what the vendors are going to make of it.. hell, they still can't get to grips with normal init.d/rc?.d scripts..

(Don't get me wrong - I like the SMF.. it's just going to be funny seeing a Solaris machine silently boot.... quickly).

Clones, Failovers and Migrations

nospam@example.com (Mike Scott) — Tue, 31 Jan 2006 06:05:44 -0500

I've recently finished my fourth article for SysAdmin magazine - this was a bit of a rush job, as they want to publish in the "Clusters" themed issue in April. It is a discussion of a technique for system failover that I recently put into production, which more-or-less guarantees transparency of failover.

I'll not publish the full article online until after the magazine put it in print, however, here's the introduction.

Whilst working at a customer site, I recently had a requirement to design a mechanism whereby a very large legacy database server could be failed over from one site to another.
Over the years several methods of doing so had been investigated, and subsequently abandoned for various financial, technical and political reasons.

Years after the initial service was implemented, the system was still a major single point of failure for the customer, and scheduled to still be in active duty for anything up to eighteen months.

During this time, the size and criticality of the system meant that the risk of implementing a creative solution was deemed to be inappropriate, until a single, significant event forced the client to reassess the situation.
The solution had to be constructed around the existing system, keeping change to a minimum, and absolutely avoiding any direct interference with the database and application.

We had a healthy budget to implement, however, we felt that a competent solution could be delivered whilst utilising a fraction of the available funds.

SAME again

nospam@example.com (Mike Scott) — Thu, 26 Jan 2006 11:33:00 -0500

Back onto the subject of striping, here's a thought that I've harboured for a little while, but never actually gotten around to testing it formally. Perhaps sometime soon I'll find the time to give it a thorough test and analysis and report back..

Common knowledge tells us to take the average size of a read or write operation (dependant on whether an app is read- or write- mostly) and to divide it by the number of spindles that we are striping against in order to calculate the correct column width.

I'm not so sure.

The only way that this mythical average operation will utilise every spindle once is if (and only if) it starts at block zero of the stripe.

Another
quick calculation here. A typical stripe width is 1MB (2048 blocks),
our chance of the operation starting at block zero is about 0.049%.

Let's also think about what happens when an "average" operation starts at block n, (where n>0).
The operation will clearly "wrap around" the stripe, and partially
write onto the very next column (which happens to be the same disk that
we started on)

There is a relatively simple solution, and that is how I belive we ought
to be calculating the column widths: Take the size of the "average"
operation, divide by the number of spindles/LUNs (and here's the
important bit).. minus one.

So for example, we have an average operation of 1MB, and eight spindles. We can calculate our desired column width by:

1024KB / (8-1) ~= 146KB

Conventional wisdom would have placed this at:

1024KB / 8 = 128KB

The
next question is likely to be "will this have a major impact?", and
therefore "do we care?". Not sure to be honest. I'd like to find the
time to fire up Dtrace for a proper test. It seems likely that using
the larger column width ought to reduce the number of operations, and
possibly then reduce contention.

Another closure, another customer gone..

nospam@example.com (Mike Scott) — Wed, 25 Jan 2006 03:15:21 -0500

With the recent news that Lexmark is closing it's inkjet cart manufaturing facility in Rosyth, I'll be losing a good customer.

It would seem that the plant has closed for economic reasons - presumably the cost of production per unit is too high. This is likely to be due to a number of things - falling prices of laser printers, increased competition between the inkjet refill suppliers, but the gaze of suspicion has to look at the manpower costs that is incurred by running a workforce in a so-called first world country.

The company also has major production facilities in the Philipines and Mexico, presumably which will be taking up the Scottish workload.

It would appear that the large corporates are taking an increasingly abstract view on their global locations, concentrating on the bottom line, and to be honest, can we really blame them? I recently shopped around to purchase a consumer electrical product at the cheapest price. In many ways, they are doing just the same, just on a rather grander scale.

Where will this lead us though? - I'm no financial or economic expert, but surely this trend is unsustainable in the long term...

SAME difference

nospam@example.com (Mike Scott) — Tue, 24 Jan 2006 03:33:00 -0500

I was chatting with Doug recently about disk layouts, and the conversation rolled onto disk and volume topologies and layouts.

Where I'm working at the moment, there has been a myth propogated that striping is good for data filesystems, and concats are better for redo logs.

Quite where this has come from, I'm not entirely sure. However.. whilst thinking about this I came to the conclusion that for redo logs, the layout is largely irrelevant.

Assuming that is, if you are striping, you are using a reasonably large column width, and you're not trying to RAID5 the redo log, which would likely result in poor performance unless accelerated by some write caching controller. The thinking went something along these lines:

The theory is that the size of the redo log write will be very small (a matter of bytes or, at the most a couple of kilobytes, I'd imagine (it's been a while since I've actually traced any of this through to physical I/O - so this is a largely theoretical argument) so let's assume for the time being that an "average" redo write is 1KB, and we have a striped layout with a column width of 256KB (this was based on a specific example), the chance of a redo log write spanning two columns (and therefore requiring two separate physical write ops instead of one) is going to be:

KB in a column= 256
Bytes in a column= 256*1024= 262144

For a two block write (2* 512bytes) to span two columns, it would require to start at least (512*2-1) bytes from the end of the column (=1023 bytes)

The probability therefore of a redo log write requiring two separate write operations = 1024/262144= 0.0039% (and even then, the two writes are on distict spindles, possibly even different controllers, and will probably happen in parallel).

Based on the aforementioned assumptions - that sort of probabilities I can live with, and so can the DBA team. So, whilst we are striping the data volumes, it makes good sense to also stripe the redo logs. Keep it the SAME.