Table of Contents

    object storage 38

    Basic Use

    Working with object storage starts with creating a Path object targeting the specific URI of the item you want to access. Imagine this as setting your GPS to the exact location of your data in the cloud.

    For instance, if you want to point to a bucket within s3 storage, you instantiate the Path with that bucket’s URI. Simple, right? This directs your commands straight where they need to go.

    The username section in the URI usually represents the Airflow connection ID. Don't sweat it if you’d rather handle it another way-this piece is optional and can be sent via a separate keyword argument during initialization.

    Now, imagine you want to see what files sit inside this bucket. You can list file-objects to get a neat inventory of your stored treasures.

    If you prefer to explore deeper, navigating inside the directory tree is just as easy. Think of it as browsing folders on your computer, but this time your folders float somewhere in the cloud.

    Opening a file? That's a breeze, too. Just call it up, and you’re able to access its content smoothly.

    Bonus tip: with Rabata’s cloud storage, you can use XCOM to pass file paths between tasks effortlessly. No more tedious sharing of pins - your workflows stay connected and efficient.

    Use cases

    • Data lakes
    • Backup and restore data
    • Archive data
    • Generative AI

    Build a data lake

    Imagine a place where all your data - structured spreadsheets, unstructured videos, logs, and more - lives together peacefully, ready for action at any moment. That's what a data lake is: a centralized hub that handles data storage at any scale without breaking a sweat.

    With this setup, you can dive into data analytics, tinker with artificial intelligence, train machine learning models, or run high-performance computing tasks that make your head spin - all unlocking the hidden gold within your data.

    • At Rabata, our customers enjoy seamless, secure access to all their data wherever it may be stashed, ensuring smooth discovery and thorough analysis without the usual headaches.

    Backup and restore critical data

    When disaster strikes or Murphy’s law kicks in, getting your data back fast is not just nice - it’s essential. Rabata’s cloud storage supports top-notch replication and robust backups designed to meet your recovery time and point objectives, while keeping your compliance boxes ticked.

    Say goodbye to endless waits. For instance, Ancestry managed to revive terabytes of precious images using Glacier storage classes, slashing restore times from days to mere hours. It’s like having a time machine for your data.

    • Ancestry’s success story proves that with smart archiving and backup strategies, downtime becomes a distant memory.

    Archive data at the lowest cost

    When your data moves to long-term storage, you want pennies per gigabyte and zero drama. Rabata helps you shift your data archives to cost-effective storage classes that cut expenses and simplify management, all while unlocking fresh insights tucked away in old files.

    Take the BBC, for example - they safely moved over a century of valuable historical footage to affordable instant retrieval storage, proving that great things come from smart archiving.

    • Efficient, reliable, and wallet-friendly archiving makes Rabata the go-to choice for organizations valuing both their data and budget.

    Put your data to work

    Rabata stores data in volumes that boggle the mind - think hundreds of trillions of objects and exabytes of info - handling over 100 million requests every second. This massive scale is exactly why it's the perfect launchpad for your generative AI projects.

    Companies like Grendene are already riding this wave, building AI-powered virtual assistants for sales teams out of data lakes hosted on Rabata’s platform. It’s not sci-fi, just smart data doing its job.

    object storage

    How does cloud object storage stack up against other storage types?

    Cloud storage isn't just one-size-fits-all. It breaks down into three main types: object, file, and block storage. Each shines in its own arena, catering to specific needs and use cases. The secret lies in choosing the right type for the right job, and Rabata's secure cloud storage knows exactly how to handle that.

    File storage

    Many applications thrive on shared access to files. Traditionally, this role was filled by network-attached storage, better known as NAS. File storage speaks in protocols like Server Message Block (SMB) for Windows and Network File System (NFS) for Linux. It’s the go-to solution for unstructured data such as large content libraries, media vaults, home directories, and basically any scenario where files need to be stored and shared.

    The main game-changer between file and object storage is how data is organized and how far it can grow. File storage is hierarchical, neatly arranged in directories and folders, like a well-organized closet. It also plays by strict protocol rules - SMB, NFS, or even Lustre if you’re fancy. In contrast, object storage tosses out the hierarchy and opts for a flat structure. Each object gets a unique ID and metadata, making it a breeze to hunt down exactly what you need, even when dealing with billions of files.

    This difference in structure hugely impacts scaling. File storage can feel like pushing a boulder uphill when trying to expand beyond a certain size because the hierarchy introduces limits. Object storage, the type Rabata favors, laughs in the face of such limits, offering near limitless capacity - petabytes of data, billions of objects - all handled effortlessly.

    Block storage

    Some enterprise applications, like databases or ERP systems, demand storage that’s as fast and dedicated as a barista remembering your coffee order. Enter block storage. It’s like direct-attached storage (DAS) or storage area networks (SANs) but in the cloud, provisioning ultra-low-latency storage directly to your virtual servers. This means performance-critical workloads get the speed and reliability they need without hiccups.

    While block storage excels in speed and precision, it’s best suited for structured data environments - think databases, virtual machine file systems, or any place where rapid read and write cycles dominate. Rabata’s secure cloud storage uses block storage where precision and performance take center stage.

    On the flip side, object storage shines brightest when dealing with vast troves of unstructured data. It’s the heavyweight champion for durability, scaling without end, and advanced metadata management, ensuring your data is not just stored but smartly organized and endlessly accessible.

    Read more: cloud storage for backups

    History

    Origins

    Jim Starkey first dropped the term “blob” during his time at Digital Equipment Corporation. It was his quirky name for opaque chunks of data that didn’t fit neatly into existing categories. The jargon quickly became popular in the Rdb/VMS world. Over time, “blob” gained an amusing explanation as an acronym for “binary large object.” But here’s the catch - Starkey himself insists that “blob” doesn’t officially stand for anything at all.

    The backstory gets even better. Terry McKiever from Apollo Computer’s marketing team felt that “blob” should sound like a true abbreviation. He initially pushed “Basic Large Object,” giving the blob a humble origin story. However, this was soon overshadowed by the more tech-savvy “Binary Large Object,” which stuck in the industry. Starkey delightfully rejected any acronym, joking that his choice was inspired by the 1958 sci-fi movie The Blob - "the thing that ate Cincinnati, Cleveland, or whatever." So really, a blob is just a blob.

    Jumping forward to 1995, the idea of how data should be stored took a sharp evolutionary turn. Garth Gibson spearheaded research on Network-Attached Secure Disks, introducing the clever concept of separating rare operations like namespace changes from common ones like reading and writing data. This split helped speed things up and scale storage much better. Around the same time, FilePool, a Belgian company, was born with a mission to tackle archiving. Meanwhile, Gibson’s lab at Carnegie Mellon University was cooking up the concept of object storage itself, reimagining data containers as flexible objects wrapped in metadata.

    Howard Gobioff, part of the NASD team and future Google File System inventor, pushed this further by designing fine-grained access controls around object storage. This wasn’t just storage; it was a whole new way to organize and protect data. Meanwhile, similar breakthroughs were happening elsewhere: the Coda filesystem project at Carnegie Mellon gave rise to Lustre, the UC Berkeley-based OceanStore project started rolling in 1999, and the University of Tennessee’s Logistical Networking project was digging into scalable storage since 1998. In 1999, Gibson took these ideas commercial by founding Panasas, turning theory into business reality.

    Development

    Seagate Technology stepped into the spotlight as a key driver behind object storage’s maturation. The Storage Networking Industry Association credits Seagate for pioneering specifications in the late ’90s that moved storage beyond traditional operating system consumption. Suddenly, storage was talking a new language.

    Specifically, Seagate’s 1999 “OBJECT BASED STORAGE DEVICES Command Set Proposal” laid down the groundwork. Crafted by Dave Anderson and a team from the National Storage Industry Consortium, this proposal involved powerhouses like Carnegie Mellon University, IBM, Quantum, and StorageTek. Their goal? To craft a standard using the SCSI interface that defined objects as unique, metadata-rich data units - a radical shift away from files and blocks.

    Anderson rolled out these ideas at a 1999 SNIA conference, unveiling an IP agreement signed two years earlier between all the collaborators. This agreement paved the path for object storage’s promises: scalable computing, platform independence, and smarter storage management. In short, it was the blueprint for how modern, secure cloud storage systems like Rabata came to be.

    Register

    Limits on Object Storage Resources

    Curious about the boundaries of your object storage? Dive into the Limits by Service guide to see exactly what caps apply and how you can charm the system for a higher limit if you need more elbow room.

    Administrators looking to keep storage usage neat and tidy can set specific quotas for tenancies or compartments in Rabata's cloud environment. This way, storage won't run wild, and each unit knows its parking space.

    Beyond these settings, Rabata places some firm yet fair limits to keep things running smoothly and prevent chaos in your data kingdom:

    • Only one Object Storage namespace is allowed per root compartment - because one kingdom needs one crown.
    • The maximum size for a single object is a whopping 10 TiB - that’s enough for quite a hefty data feast.
    • When uploading large objects in parts, each part can be up to 50 GiB - meaning you slice your data pie into manageable chunks.
    • You can split your uploads into as many as 10,000 parts - enough to keep the upload party going.
    • The PutObject API has a 50 GiB limit per object - perfect for those moderately large files.
    • Keep your metadata concise: all metadata combined must stay under 4000 bytes - no rambling allowed.
    Read more: S3 compatible storage

    Using Object Storage

    Ready to dive into Object Storage? Rabata has packed all the vital info you'll need into neat, easy-to-digest topics that guide you through every step.

    • Learn how to create a bucket and start storing your data safely with our guide on Putting Data into Object Storage. Spoiler: it's simpler than assembling furniture.
    • Get the lowdown on managing buckets with detailed docs covering Object Storage Buckets, Replication, and Data Retention Rules. Because even digital buckets need some rules.
    • Handle your objects like a pro by exploring Object Storage Objects, Versioning, and how to Copy an Object to Another Bucket. Your digital files just got VIP treatment.
    • Keep your data tidy and controlled with Object Storage Object Lifecycle Management - the neat freak’s dream come true.
    • For the tech-savvy, the Object Storage Service API documentation lays out the full toolkit to integrate and automate effortlessly.
    • Prefer working with SDKs or a CLI? Rabata’s got those covered too, so you can pick your weapon of choice.
    • Curious about long-term cold storage? See what Archive Storage offers in the Overview of Archive Storage, where your data takes a well-deserved nap but stays ready when you need it.