Sunday, January 16, 2011

Deleting files in Linux

I need to find all files matching a certain criteria and delete them - here's a snippet:

/var/www/somesite/releases/{many directories}/tmp/attachment_fu

I'd like to find all files in any tmp/attachment_fu directory and delete them - the problem is that the {many directories} is throwing off my find skills (or maybe find is the wrong command - I also tried locate to no avail).

  • I'm not completely clear on what you're trying to do - delete files matching the name attachment_fu recursively, but leaving directories alone?

    If so, try this (I've added an echo so you can test-drive it first)...

    find /var/www/somesite/releases -type f -name attachment_fu -exec echo rm -f {} \;
    

    If not, please explain further :)

    Alternatively, you can use the -regex flag to find if the '/tmp/' is important; something like...

    find /var/www/somesite/releases -type f -regex '.*/tmp/attachment_fu$' -exec echo rm -f {} \;
    

    All this is assuming that the files you're after are at various depths in the filesystem tree, otherwise you can just use rm (post by radius).

    From Xerxes
  • May be I'm missing something but rm /var/www/somesite/releases/*/tmp/attachment_fu/* seems to be what you want. ls /var/www/somesite/releases/*/tmp/attachment_fu can be used to see what will be deleted

    Dennis Williamson : attachment_fu is a directory, so you'll need another asterisk at the end.
    : As Dennis mentioned, a slightly modified version of this worked: `rm /var/www/somesite/releases/*/tmp/attachment_fu/*`
    Lee B : Any and all files also means hidden files. Something like rm -rf /var/www/somesite/releases/*/tmp/attachment_fu/{,.}* is probably what you want. Be careful, as -rf makes this a pretty dangerous command. Put ls in front and |less on the end if you want to test it. p.s.: if you were doing it with find, you'd probably want to limit the depth with min-depth and max-depth options.
    radius : Thanks Dennis, I corrected the missing *
    From radius
  • ls /var/www/somesite/releases/*/tmp/attachment_fu | xargs rm -rf

    something like that?

    yeah, or rm -rf /var/www/somesite/releases/*/tmp/attachment_fu

    ;)

    radius : He's talking about removing file, not directory, so -rf il probably not what he wants !
    From
  • rm -rf said path it will remove all dir and files

    reinierpost : The question is how to only remove files matchign a certain pattern.
    From Rajat
  • In addition to other answers, you can incur in the "command line too long" error if using * when the number of files you have to delete is huge.

    The safest way could be to use the shell expansion to retrieve the list of dirs, loop them and use find to remove the files:

    for d in $(ls /var/www/somesite/releases/*/tmp/attachment_fu)
    do
      find $d -type f -exec rm {} \;
    done
    

    This should work quite well, of course it would fail if you have an huge number of dirs, but then you have other kind of problems :P

    From AlberT

Apache gives empty reply

It happens randomly, and only on moodle installations. Apache don't add a line in the logs when this happens, and I don't know where to look.

koke@escher:~/Code/eboxhq/moodle[master]$ curl -I http://training.ebox-technologies.com/login/signup.php?course=WNA001 
curl: (52) Empty reply from server
koke@escher:~/Code/eboxhq/moodle[master]$ curl -I http://training.ebox-technologies.com/login/signup.php?course=WNA001 
HTTP/1.1 200 OK

The apache conf is quite straightforward and works perfectly in the other vhosts

<VirtualHost *:80>
    ServerAdmin webmaster@ebox-technologies.com
    DocumentRoot /srv/apache/training.ebox-technologies.com/htdocs
    ServerName training.eboxhq.com
    ErrorLog /var/log/apache2/training.ebox-technologies.com-error.log
    CustomLog /var/log/apache2/training.ebox-technologies.com-access.log combined

        <FilesMatch "\.(ico|gif|jpe?g|png|js|css)$">
                ExpiresActive On
                ExpiresDefault "access plus 1 week"
                Header add Cache-Control public
        </FilesMatch>
</VirtualHost>

Using apache 2.2.9 php 5.2.6 and moodle 1.9.5+ (Build: 20090722)

Any ideas welcome :)

  • Is there nothing in error_log either?

    You could set MaxServers to 1, and then strace that apache thread while requesting the page. It could be that PHP is segfaulting, or the apache child is dying for other reasons.

    Also, tcpdump the request at both ends to see what data is being sent on the wire.

  • Look at the main server error log, it should have some Child Segfault Error (11).

    This is probably given by a PHP engine fault, often given by a buggy php module, unicode chars handling, gettext corrupted DB, and so on.

    It is very difficult to debug.

    You can enable xdebug tracing and look at what line it is segfaulting:

    pecl install xdebug
    

    php.ini:

    [xdebug]
    xdebug.profiler_enable=1
    xdebug.profiler_output_dir="/tmp/xdebug"
    xdebug.auto_trace=1            ; enable tracing
    xdebug.trace_format=0
    xdebug.show_mem_delta=1        ; memory difference
    xdebug.show_local_vars=1
    xdebug.max_nesting_level=100
    
    Jorge Bernal : Found this on error log: zend_mm_heap corrupted [Fri Sep 18 16:47:08 2009] [notice] child pid 7361 exit signal Segmentation fault (11) [Fri Sep 18 16:47:09 2009] [notice] child pid 7368 exit signal Segmentation fault (11) [Fri Sep 18 16:47:10 2009] [notice] child pid 7375 exit signal Segmentation fault (11) Will try xdebug and see if it helps
    Jorge Bernal : Only see the segmentation fault and xdebug is not leaving anything useful in /tmp/xdebug
    AlberT : it should print a trace of _every_ php function called. So when the trace stops is when the segfault happened. This should suffice to argue what php function is faulting.
    From AlberT
  • Finally solved by adding to /etc/apache2/envvars:

    export USE_ZEND_ALLOC=0
    
  • Finally solved by adding to /etc/apache2/envvars:

    export USE_ZEND_ALLOC=0

    followe dur steps... Now my webmail works again!

    thanks!

    From dave yazz

Friday, January 14, 2011

block level vs. file level cloning?

I've always been a block level kinda guy but I'm interested in hearing some real world experiences with file level cloning. What are some of the advantages and disadvantages as well as what tools work the best.

  • Well the most obvious advantage of file-level cloning is that you don't waste time cloning unused blocks. Eg a clone of a 40G partition with 10G of data will require 40G of reads and 40G of writes on the block level, but close to 10G of reads and 10G of writes on the file level.

    One minor benefit of file-level cloning, is that it effectively perfectly de-fragments your filesystem at the same time, whereas block-level cloning clones fragmentation as well.

    Block-level cloning is simpler, and you don't have to worry about any kind of permissions or other issues, you know for 100% certain the clone will be identical to the original, but it's possible for file-level cloning to go wrong if you mess up your settings.

    Mark : If you have a tool to do block level cloning that is aware of the filesystem structure, it can skip unused blocks, and so it may not need the full 40GB of reads/writes. I'm thinking zfs send/receive here, but I'm sure there are other filesystems or tools that do something similar.
    womble : If it's aware of the filesystem structure, it's a file-level cloning tool.
    From davr
  • My worst experience with file level cloning was a 20Gig NT4 partition with about 1.6m tiny files. Transfer rate would have been ~8Meg/sec with block level cloning (over a 100Meg network) and should have taken somewhere between an hour and an hour and a half, it ended up at <150K/sec because of all of the file system\permissions overhead and took almost two days.

    From Helvick
  • As people have said use block level when the hit on the file meta data is too great. Use file based when there are not many files.

    I am used to a block replication system that only replicates blocks that have changed and are allocated to files. This can work very well.

    File based replication is cheap and easy to do on an open system however rsync/unison scripts need more maintenance than replication on a NAS or a SAN.

    If there are millions of files then block level is the only way to go, we have a number of filesystems that have 40 million files in 600GB and file based replication is not going to work there.

    From James

TrueCrypt or EFS?

A subset of my users need a way to share an encrypted folder on the file server. Security is the most important, followed closely by ease of use. It appears that TrueCrypt is easier to set up. Does EFS have any advantages over TC to justify the extra setup?

Windows Server 2003 and XP, Active Directory, 100 user LAN.

Edit: I originally missed the limitation of single-user R/W access for Truecrypt. Looks like EFS is better once I get past the setup.

  • The section on Sharing over a Network from the TrueCrypt user's guide makes it look like you have a couple of solutions-- mounting the shared file hosting the volume locally on computers or mounting the file hosting the volume on the server computer. The big difference between the two is that the volume's contents will be accessible read-write to all client computers when it's mounted on the server computer and shared (albeit access to the data will cross the wire "in the clear") versus the volume being mounted read-only on all computers when mounted locally on each machine.

    If your users need seamless read/write access to the encrypted files either a TrueCrypt server-side mount or EFS is probably a better choice. The data is still going to cross the wire in the clear with EFS, as with TrueCrypt and the server-side mount.

    Some people get really down on EFS but I think it fills a niche and solves a problem. It's well designed for what it is, but the problem that it seeks to solve is fundamentally awkward to solve.

    Configuring EFS in an AD envrionment really isn't too difficult to setup. The most difficult part is wrapping your mind around the recovery agent functionality and exporting the recovery key to a safe offline location. You will need a PKI, but Microsoft's Certificate Services can automate most of the process for issuing certificates to users (have a look here for information about autoenrollment in Windows XP: http://technet.microsoft.com/en-us/library/bb456981.aspx)

    Have some a look at the docs from Microsoft: http://technet.microsoft.com/en-us/library/cc962122.aspx (and another at http://technet.microsoft.com/en-us/library/bb457116.aspx)

    Multi-user access to EFS files is a bit of a "wart" on the part of Microsoft, but it's not too hard to deal with. There's a very good answer here re: multi-user access to EFS-encrypted files.

    Nathan Hartley : I thought I read that data to and from an EFS share WAS encrypted. [time passes] Ah! It is encrypted on-the-wire when ran in WebDav mode... Remote EFS Operations on File Shares and Web Folders http://technet.microsoft.com/en-us/library/bb457116.aspx#EHAA
    Jim B : on a side note if you want all traffic encrypted you simply need to enable domain isolation.
    Evan Anderson : @JimB: You're absolutely right in the sense that you should use an over-the-wire encryption mechanism, such as IPsec, if you want over-the-wire encryption of any data, EFS-stored or otherwise. The "domain isolation" term was always one that rubbed me the wrong way-- sounded like a marketing-ism.
    Jim B : @Evan- It's a pretty accurate term. If you are not a domain menber you can't see any traffic in that domain. There are a couple of downloadable labs on the technet site to play with it.
  • EFS will allow you to use your existing AD and kerberos credentials to access the encrypted data.

    Truecrypt doesn't support multi-user access, and has no way of storing access credentials in a directory. Additionally, Truecrypt hasn't been FIPS 140-2 validated, so if you are encrypting to protect yourself against breaches of personally identifying information it isn't the right tool.

    Also consider commercial products like McAfee File & Folder encryption.

  • I'd recommend going with TrueCrypt for this scenario. Its probably going to be easier than EFS in this case.

    From KPWINC

Retrieve operational attributes from OpenLDAP

I've been having trouble trying to find some good documentation on how to retrieve operational attributes from OpenLDAP.

I would like to retrieve the base distinguished name of an LDAP server by doing an LDAP search.

How come my search doesn't work when I explicitly ask for namingContexts attribute? I've been told that I need to add a plus ('+') sign to the attribute list.

If this is the case, should I get rid of the "namingContexts" attribute or have both?

ldapsearch -H ldap://ldap.mydomain.com -x -s base -b "" +
# note the + returns operational attributes

Edit: Note how it looks like the attributes requested are empty. Shouldn't the plus sign be in the attribute list? http://www.zytrax.com/books/ldap/ch3/#operational

reference: plus sign operator with OpenLDAP

  • How come my search doesn't work when I explicitly ask for namingContexts attribute?

    What is not working? Do you recieve an error?

    When there is a plus sign it returns all the attributes, regardless if namingContexts is added.

    Using:

    ldapsearch -x -H ldap://ldap.example.com -s base -b "" namingContexts
    

    Returns:

    # extended LDIF
    #
    # LDAPv3
    # base <> with scope baseObject
    # filter: (objectclass=*)
    # requesting: namingContexts 
    #
    
    #
    dn:
    namingContexts: o=example.com
    
    # search result
    search: 2
    result: 0 Success
    
    # numResponses: 2
    # numEntries: 1
    

    It is also listed using:

    ldapsearch -x -H ldap://ldap.example.com -s base -b "" +
    

    Returning:

    # extended LDIF
    #
    # LDAPv3
    # base <> with scope baseObject
    # filter: (objectclass=*)
    # requesting: + 
    #
    
    #
    dn:
    structuralObjectClass: OpenLDAProotDSE
    namingContexts: o=example.com
    supportedControl: 2.16.840.1.113730.3.4.18
    supportedControl: 2.16.840.1.113730.3.4.2
    supportedControl: 1.3.6.1.4.1.4203.1.10.1
    supportedControl: 1.2.840.113556.1.4.1413
    supportedControl: 1.2.840.113556.1.4.1339
    supportedControl: 1.2.840.113556.1.4.319
    supportedControl: 1.2.826.0.1.334810.2.3
    supportedExtension: 1.3.6.1.4.1.1466.20037
    supportedExtension: 1.3.6.1.4.1.4203.1.11.1
    supportedExtension: 1.3.6.1.4.1.4203.1.11.3
    supportedFeatures: 1.3.6.1.4.1.4203.1.5.1
    supportedFeatures: 1.3.6.1.4.1.4203.1.5.2
    supportedFeatures: 1.3.6.1.4.1.4203.1.5.3
    supportedFeatures: 1.3.6.1.4.1.4203.1.5.4
    supportedFeatures: 1.3.6.1.4.1.4203.1.5.5
    supportedLDAPVersion: 2
    supportedLDAPVersion: 3
    supportedSASLMechanisms: DIGEST-MD5
    supportedSASLMechanisms: CRAM-MD5
    subschemaSubentry: cn=Subschema
    
    # search result
    search: 2
    result: 0 Success
    
    # numResponses: 2
    # numEntries: 1
    
    From
  • What version of OpenLDAP are you using? What does "doesn't work" mean precisely? What is the output when you run that command?

    I ran it on my OpenLDAP instance and it produced output similar to carrell's.

    I'm wondering if it may be a permissions issue. Perhaps anonymous users don't have read access on dn="" or access to the operational attributes in question?

  • First access rule in my slapd.conf is explicitly to make sure that this is permitted; make sure you have something similar:

    # Let all clients figure out what auth mechanisms are available, determine
    # that TLS is okay, etc
    access to dn.base=""
            by *            read
    
    From Phil P

Positive vs. negative monitoring

Ive been looking at monitoring for a while. My org didnt have any before i came other than 'whered my yahoo go'. It appears that most packages out there focus on negative monitoring (ie, this service/host was up and now its not). This seems like a valid first step, but what can you look at past that for positive monitoring (ie that port wasnt up, and now it is, or hey look thats a new DHCP host)? I suppose its possible to have a declaration for every single port/network address in nagios, but that seems cumbersome.

Does anyone know of a better tool for monitoring ports/hosts for affirmatively down?

  • Nagios includes a wide range of plugins and modules for active/intrusive monitoring and passive monitoring. It should include everything you need!

    From Aiden Bell
  • What you are looking for isn't really monitoring as much as it is security. I am not a security expert, but there are a number of network scanning tools out there that can be "taught" what to expect and then will tell you if something is out of the ordinary.

  • For hosts that you know about, Nagios/Zenoss/OpenNMS are your best bet - they can be configured to notify when hosts and/or services go down, or come back up. They're mostly smart enough not to start alerting about ALL the services on a host if the host itself's down, as well; it's important to configure these sorts of things properly, so that you don't get deluged with 20 alerts because of a server reboot. If there's that much information about trivial stuff, sooner or later you'll end up pretty much ignoring it and missing something important.

    For the second half of your question, Catherine's right; you're looking at an Intrusion Detection System (IDS). These can be configured to know what your network should look like in terms of hosts, topology, traffic types and so on, then alert you if anything other than what you've defined as "ordinary" happens. A couple of examples would be Snort and OSSEC.

    From RainyRat
  • We use nmap for this. We have a simple script wrapping nmap that scans our entire network and stores the XML output. The next night it runs again and compares the output. If any new hosts or ports show up, an email is sent to the admin staff.

    The just-released Nmap 5.0 includes a utility for just this purpose called Ndiff.

    From Insyte
  • For your specific questions, I'd use something like arpwatch to watch for changes in ARP addresses and portsentry to watch for anyone trying to connect to unused ports. You could use other tools as well.

    These tools can then be integrated into an active or passive check for Nagios.

    Saurabh Barjatiya : arpwatch will work only for subnet in which host is running and portsentry is probably for protecting individual host when it detects port scan.
    From David

mobileadmin with exchange 2003

We have several users connected to our exchange server using activesync using a variety of devices. The documentation for mobileadmin appears to indicate that the app should provide a list of active devices somewhere, but i cant seem to find it. Is there a config change i need to make, or are the docs inaccurate?

  • I think are misreading it. you get:

    View a list of all devices that are being used by any enterprise user

    Select/De-select devices to be remotely erased

    View the status of pending remote erase requests for each device

    View a transaction log that indicates which administrators have issued remote erase commands, in addition to the devices those commands pertained to

    ANervousTwitch : thats probably correct, since thats the functionality thats there now. it seems easily read to mean a list of all devices though.
    From Jim B