Unix Solaris Linux – Directory failing to backup – ‘ls’ error ‘not enough memory’

Unix Solaris Linux – Directory failing to backup – ‘ls’ error ‘not enough memory’

Unix Solaris Linux – Directory failing to backup
‘ls’ error ‘not enough memory’

Directory Failing to Backup

Currently we use Netbackup to backup our systems a we started to a failure message on one of our Oracle directories on an mature Solaris server.
I remote shelled into the server, changed to the offending directory (/u01/oracle/admin/DIM/adump) and ran ls but it came backĀ  with a ‘not enough memory’ error. I should have run the ls command with the -u flag (unsorted) to see how many files existed in the directory but I knew it was a lot.

The adump (audit) directory

I have Burleson Consulting to be a good source of Oracle SQL Server information and on his webpage it give information about the location and the configuration of the adump directory.
Our system has been dumping audit files into this directory at a rate of a few hundred a day.SQL> show parameter audit.

NAME TYPE VALUE
———————– ———– ————–
audit_file_dest string /u01/app/oracle/admin/DIM/audit
audit_sys_operations boolean TRUE
audit_trail string DB

Find – a dangerous command with -exec rm -f {} \;

I knew we had a massive backlog of files which need to be deleted I used the find command to see how many files over ten years existed assuming a year is 365 days. Unix stores three dates for each file, creation, modified and accessed ‘date and time’.
I used the modified date to identify if we had files older than ten years with the shell in the audit directory (I used the full path as I’m paranoid). I also used a filter to only select files ending in ‘aud’ as I was going to use this command to delete files:

find /u01/oracle/admin/DIM/adump/ -name “*.aud” -mtime +3650 -ls

There was quite a few audit files over ten years old so I ran the rm command with the help of find:

find /u01/oracle/admin/DIM/adump/ -name “*.aud” -mtime +3650 -exec -f {} \; -ls

This would find files in the directory ending in ‘aud’ which were older than ten year, delete them and then list the file which was being deleted. Running the command initially without the rm and then adding it and running with a few files at a time as this could create a serious issue if it was let loose deleting files.
I ran it against a year at a time which took several hours per year. There was also the risk of freeing up storage on the disk subsystem which hadn’t been written to in over ten years.
I removed five years of audit files as this would allow the backup to run if we need to and as the system will be upgraded soon I won’t have to worry about the number of files.

Unix Solaris Linux – Directory failing to backup – ‘ls’ error ‘not enough memory’

About 10 million audit files

I would appear we had about 10 million audit files and we down to about half.

Unix - ls | wc -l - this lists the files and then does a word count.