java hadoop file system API

欲忘树

org.apache.hadoop.fs
Class FileSystem

• java.lang.Object
  
• org.apache.hadoop.fs.FileSystem
  

• All Implemented Interfaces:Closeable, AutoCloseable, ConfigurableDirect Known Subclasses:FilterFileSystem, FTPFileSystem, KosmosFileSystem, NativeS3FileSystem, RawLocalFileSystem, S3FileSystem, ViewFileSystem
  @InterfaceAudience.Public
  @InterfaceStability.Stable
  public abstract class FileSystem
  extends Configured
  implements Closeable
  
  An abstract base class for a fairly generic filesystem. It may be implemented as a distributed filesystem, or as a "local" one that reflects the locally-connected disk. The local version exists for small Hadoop instances and for testing.　　All user code that may potentially use the Hadoop Distributed File System should be written to use a FileSystem object. The Hadoop DFS is a multi-machine system that appears as a single disk. It's useful because of its fault tolerance and potentially very large capacity.
  　　The local implementation is LocalFileSystem and distributed implementation is DistributedFileSystem.
  
  

• 
  
  
  
  • 
    Field Summary
    
    Fields
    
    Modifier and TypeField and Description
    
    static String
    DEFAULT_FS
    
    
    static String
    FS_DEFAULT_NAME_KEY
    
    
    static org.apache.commons.logging.Log
    LOG
    
    
    static int
    SHUTDOWN_HOOK_PRIORITY
    
    Priority of the FileSystem shutdown hook.
    
    
    
    protected org.apache.hadoop.fs.FileSystem.Statistics
    statistics
    
    The statistics for this file system.
    
    
    
  
  
  
  
  • 
    Constructor Summary
    
    Constructors
    
    ModifierConstructor and Description
    
    protected
    FileSystem()
    
    
  
  
  
  
  • 
    Method Summary
    
    [table]Methods
    
    [tr]Modifier and TypeMethod and Description[/tr]
    [tr]
    [td]FSDataOutputStream[/td]
    [td]append(Path f)
    
    Append to an existing file (optional operation).
    [/td]
    [/tr]
    [tr]
    [td]FSDataOutputStream[/td]
    [td]append(Path f, int bufferSize)
    
    Append to an existing file (optional operation).
    [/td]
    [/tr]
    [tr]
    [td]abstract FSDataOutputStream[/td]
    [td]append(Path f, int bufferSize, Progressable progress)
    
    Append to an existing file (optional operation).
    [/td]
    [/tr]
    [tr]
    [td]boolean[/td]
    [td]cancelDeleteOnExit(Path f)
    
    Cancel the deletion of the path when the FileSystem is closed
    [/td]
    [/tr]
    [tr]
    [td]protected void[/td]
    [td]checkPath(Path path)
    
    Check that a Path belongs to this FileSystem.
    [/td]
    [/tr]
    [tr]
    [td]static void[/td]
    [td]clearStatistics()
    
    Reset all statistics for all file systems
    [/td]
    [/tr]
    [tr]
    [td]void[/td]
    [td]close()
    
    No more filesystem operations are needed.
    [/td]
    [/tr]
    [tr]
    [td]static void[/td]
    [td]closeAll()
    
    Close all cached filesystems.
    [/td]
    [/tr]
    [tr]
    [td]static void[/td]
    [td]closeAllForUGI(org.apache.hadoop.security.UserGroupInformation ugi)
    
    Close all cached filesystems for a given UGI.
    [/td]
    [/tr]
    [tr]
    [td]void[/td]
    [td]completeLocalOutput(Path fsOutputFile, Path tmpLocalFile)
    
    Called when we're all done writing to the target.
    [/td]
    [/tr]
    [tr]
    [td]void[/td]
    [td]concat(Path trg, Path[] psrcs)
    
    Concat existing files together.
    [/td]
    [/tr]
    [tr]
    [td]void[/td]
    [td]copyFromLocalFile(boolean delSrc, boolean overwrite, Path[] srcs, Path dst)
    
    The src files are on the local disk.
    [/td]
    [/tr]
    [tr]
    [td]void[/td]
    [td]copyFromLocalFile(boolean delSrc, boolean overwrite, Path src, Path dst)
    
    The src file is on the local disk.
    [/td]
    [/tr]
    [tr]
    [td]void[/td]
    [td]copyFromLocalFile(boolean delSrc, Path src, Path dst)
    
    The src file is on the local disk.
    [/td]
    [/tr]
    [tr]
    [td]void[/td]
    [td]copyFromLocalFile(Path src, Path dst)
    
    The src file is on the local disk.
    [/td]
    [/tr]
    [tr]
    [td]void[/td]
    [td]copyToLocalFile(boolean delSrc, Path src, Path dst)
    
    The src file is under FS, and the dst is on the local disk.
    [/td]
    [/tr]
    [tr]
    [td]void[/td]
    [td]copyToLocalFile(boolean delSrc, Path src, Path dst, boolean useRawLocalFileSystem)
    
    The src file is under FS, and the dst is on the local disk.
    [/td]
    [/tr]
    [tr]
    [td]void[/td]
    [td]copyToLocalFile(Path src, Path dst)
    
    The src file is under FS, and the dst is on the local disk.
    [/td]
    [/tr]
    [tr]
    [td]static FSDataOutputStream[/td]
    [td]create(FileSystem fs, Path file, FsPermission permission)
    
    create a file with the provided permission The permission of the file is set to be the provided permission as in setPermission, not permission&~umask It is implemented using two RPCs.
    [/td]
    [/tr]
    [tr]
    [td]FSDataOutputStream[/td]
    [td]create(Path f)
    
    Create an FSDataOutputStream at the indicated Path.
    [/td]
    [/tr]
    [tr]
    [td]FSDataOutputStream[/td]
    [td]create(Path f, boolean overwrite)
    
    Create an FSDataOutputStream at the indicated Path.
    [/td]
    [/tr]
    [tr]
    [td]FSDataOutputStream[/td]
    [td]create(Path f, boolean overwrite, int bufferSize)
    
    Create an FSDataOutputStream at the indicated Path.
    [/td]
    [/tr]
    [tr]
    [td]FSDataOutputStream[/td]
    [td]create(Path f, boolean overwrite, int bufferSize, Progressable progress)
    
    Create an FSDataOutputStream at the indicated Path with write-progress reporting.
    [/td]
    [/tr]
    [tr]
    [td]FSDataOutputStream[/td]
    [td]create(Path f, boolean overwrite, int bufferSize, short replication, long blockSize)
    
    Create an FSDataOutputStream at the indicated Path.
    [/td]
    [/tr]
    [tr]
    [td]FSDataOutputStream[/td]
    [td]create(Path f, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress)
    
    Create an FSDataOutputStream at the indicated Path with write-progress reporting.
    [/td]
    [/tr]
    [tr]
    [td]abstract FSDataOutputStream[/td]
    [td]create(Path f, FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress)
    
    Create an FSDataOutputStream at the indicated Path with write-progress reporting.
    [/td]
    [/tr]
    [tr]
    [td]FSDataOutputStream[/td]
    [td]create(Path f, FsPermission permission, EnumSet flags, int bufferSize, short replication, long blockSize, Progressable progress)
    
    Create an FSDataOutputStream at the indicated Path with write-progress reporting.
    [/td]
    [/tr]
    [tr]
    [td]FSDataOutputStream[/td]
    [td]create(Path f, FsPermission permission, EnumSet flags, int bufferSize, short replication, long blockSize, Progressable progress, org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt)
    
    Create an FSDataOutputStream at the indicated Path with a custom checksum option
    [/td]
    [/tr]
    [tr]
    [td]FSDataOutputStream[/td]
    [td]create(Path f, Progressable progress)
    
    Create an FSDataOutputStream at the indicated Path with write-progress reporting.
    [/td]
    [/tr]
    [tr]
    [td]FSDataOutputStream[/td]
    [td]create(Path f, short replication)
    
    Create an FSDataOutputStream at the indicated Path.
    [/td]
    [/tr]
    [tr]
    [td]FSDataOutputStream[/td]
    [td]create(Path f, short replication, Progressable progress)
    
    Create an FSDataOutputStream at the indicated Path with write-progress reporting.
    [/td]
    [/tr]
    [tr]
    [td]boolean[/td]
    [td]createNewFile(Path f)
    
    Creates the given Path as a brand-new zero-length file.
    [/td]
    [/tr]
    [tr]
    [td]FSDataOutputStream[/td]
    [td]createNonRecursive(Path f, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress)
    
    Deprecated.
    
    API only for 0.20-append
    [/td]
    [/tr]
    [tr]
    [td]FSDataOutputStream[/td]
    [td]createNonRecursive(Path f, FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress)
    
    Deprecated.
    
    API only for 0.20-append
    [/td]
    [/tr]
    [tr]
    [td]FSDataOutputStream[/td]
    [td]createNonRecursive(Path f, FsPermission permission, EnumSet flags, int bufferSize, short replication, long blockSize,Progressable progress)
    
    Deprecated.
    
    API only for 0.20-append
    [/td]
    [/tr]
    [tr]
    [td]boolean[/td]
    [td]delete(Path f)
    
    Deprecated.
    
    Use delete(Path, boolean) instead.
    [/td]
    [/tr]
    [tr]
    [td]abstract boolean[/td]
    [td]delete(Path f, boolean recursive)
    
    Delete a file.
    [/td]
    [/tr]
    [tr]
    [td]boolean[/td]
    [td]deleteOnExit(Path f)
    
    Mark a path to be deleted when FileSystem is closed.
    [/td]
    [/tr]
    [tr]
    [td]boolean[/td]
    [td]exists(Path f)
    
    Check if exists.
    [/td]
    [/tr]
    [tr]
    [td]static FileSystem[/td]
    [td]get(Configuration conf)
    
    Returns the configured filesystem implementation.
    [/td]
    [/tr]
    [tr]
    [td]static FileSystem[/td]
    [td]get(URI uri, Configuration conf)
    
    Returns the FileSystem for this URI's scheme and authority.
    [/td]
    [/tr]
    [tr]
    [td]static FileSystem[/td]
    [td]get(URI uri, Configuration conf, String user)
    
    Get a filesystem instance based on the uri, the passed configuration and the user
    [/td]
    [/tr]
    [tr]
    [td]static List[/td]
    [td]getAllStatistics()
    
    Return the FileSystem classes that have Statistics
    [/td]
    [/tr]
    [tr]
    [td]long[/td]
    [td]getBlockSize(Path f)
    
    Deprecated.
    
    Use getFileStatus() instead
    [/td]
    [/tr]
    [tr]
    [td]protected URI[/td]
    [td]getCanonicalUri()
    
    Resolve the uri's hostname and add the default port if not in the uri
    [/td]
    [/tr]
    [tr]
    [td]ContentSummary[/td]
    [td]getContentSummary(Path f)
    
    Return the ContentSummary of a given Path.
    [/td]
    [/tr]
    [tr]
    [td]long[/td]
    [td]getDefaultBlockSize()
    
    Deprecated.
    
    use getDefaultBlockSize(Path) instead
    [/td]
    [/tr]
    [tr]
    [td]long[/td]
    [td]getDefaultBlockSize(Path f)
    
    Return the number of bytes that large input files should be optimally be split into to minimize i/o time.
    [/td]
    [/tr]
    [tr]
    [td]protected int[/td]
    [td]getDefaultPort()
    
    Get the default port for this file system.
    [/td]
    [/tr]
    [tr]
    [td]short[/td]
    [td]getDefaultReplication()
    
    Deprecated.
    
    use getDefaultReplication(Path) instead
    [/td]
    [/tr]
    [tr]
    [td]short[/td]
    [td]getDefaultReplication(Path path)
    
    Get the default replication for a path.
    [/td]
    [/tr]
    [tr]
    [td]static URI[/td]
    [td]getDefaultUri(Configuration conf)
    
    Get the default filesystem URI from a configuration.
    [/td]
    [/tr]
    [tr]
    [td]BlockLocation[][/td]
    [td]getFileBlockLocations(FileStatus file, long start, long len)
    
    Return an array containing hostnames, offset and size of portions of the given file.
    [/td]
    [/tr]
    [tr]
    [td]BlockLocation[][/td]
    [td]getFileBlockLocations(Path p, long start, long len)
    
    Return an array containing hostnames, offset and size of portions of the given file.
    [/td]
    [/tr]
    [tr]
    [td]FileChecksum[/td]
    [td]getFileChecksum(Path f)
    
    Get the checksum of a file.
    [/td]
    [/tr]
    [tr]
    [td]abstract FileStatus[/td]
    [td]getFileStatus(Path f)
    
    Return a file status object that represents the path.
    [/td]
    [/tr]
    [tr]
    [td]static Class