Page 7 of 7 FirstFirst ... 5 6 7
Results 91 to 96 of 96

Thread: redundancy...

  1. #91
    Moderator VJ's Avatar
    Join Date
    Mar 2001
    Location
    Belgium/Poland/Spain
    Posts
    9,392

    Default

    Reviving this thread as my stay in Spain is nearing its end (just half a year to go... amazing how time flies) and I'll soon be back in Warsaw taking care of organizing my data properly.


    Umfriend: I have some questions regarding Drive Pool. Is it possible to do something like this:

    2 x 2TB harddisk (let's call them 2a and 2b), 1x 4TB harddisk (let's call it 4), 1x 8 TB harddisk (let's call it 8)

    Duplicate a folder on 2a + 4 + 8 and another folder on 2a + 8 (I would do something similar with 2b+4+8, 2b+8 and 4+8). If the 8 TB disk is slower does it affect the overal write speed? If so, is there a way of making it duplicate in the background? I think the plugin "SSD optimizer" seems to allow for that, or maybe the "Ordered file placement". Would this work?

    Can you set a network mount as a harddisk, or does this need CloudDrive? What happens when it is not available? (thinking of using my old computer to hold some duplicates, but it may be just easier to use some synchronization rather than DrivePool).

    Thanks!
    pixar
    Dream as if you'll live forever. Live as if you'll die tomorrow. (James Dean)

  2. #92
    Super MURCer UtwigMU's Avatar
    Join Date
    Jul 2002
    Location
    Sin City
    Posts
    4,960

    Default

    I bought camere before drives (have case, mobo, cpu, RAM) and then got hit with taxes, car service, so this project is still on a shelf.

  3. #93
    Super MURCer Evildead666's Avatar
    Join Date
    Mar 2002
    Location
    Paris, France
    Posts
    3,026

    Default

    Looks like I'm in the right place, just one year late

    I built a NAS in May-June this year. Had been thinking about it for quite some time (about a year), having data spread over multiple hard disks in a mini-itx sff case.
    I wanted some sort of redundancy in case a HDD failed, and was really wanting something to counter bit-rot as well, but that got put aside.
    I looked at FreeNAS, NAS4Free, Windows Server, Linux, and also ZFS, BTRFS, ReiserFS, CoW, all of that......
    ZFS - Requires ECC RAM, and Lots of it, which also requires Server grade Mobo, etc, Costs were too high for a home system.
    BTRFS - not stable in RAID5/6.

    Ended up making my own Synology NAS, with XPenology.
    I had an ASUS Itx board in the SFF PC, with Win7. Re-used the Asus P8H77-I (6xSata), the i3-2120, and a single 4GB DDR3 stick (the other one died...).
    Bought a Silverstone DS380 case, I liked the 8 3.5in drive bays, with 4x2.5in bays also.
    I've got 4x4TB WD RED drives in there now, used the western digital diag tool to check the drives on arrival as the shipping from every company I bought them from was terrible. One was DOA. Exchanged within 24Hrs thankfully.
    I looked at the Synology Raid FS, and BTRFS Raid, which this is capable of, but the unstable nature of BTRFS with any type of RAID 5/6 put me off that, and the Synology Raid was very user friendly, as you can add disks of varying sizes to the Raid.
    In the end I went with a simple RAID6 with EXT4, so two disks can fail. It leaves me with a little over 7TB of storage.
    I also added a 120GB SSD as read cache (just because i could. Don't need it at all).
    The XPenology boots from a USB drive, and is installed on a dedicated SSD.
    The 4 NAS drives are connected on the 4xIntel SATA ports, and the two SSD's are on the additional ASMedia SATA controller.
    I also have a relatively cheap 4xSATA controller installed to add another 4 drives down the road.
    The i3 has the stock Intel cooler i got with a faster i5.

    Basically I get 100MB/s reads and writes to it, it maxes out the GBit LAN no problem.
    It did end up costing a little bit, but i'm very very happy with it, and you control it through the Web interface.
    All I bought were the drives (the biggest cost at 4x150€) the DS380 case, ~200€, an SFF 450W PSU (dinky little thing), and the PCIe SATA controller, 50€.

    As for triple backups, i have a SCSI GigaMO back in my main PC, its a Low Profile Adaptec PCI card, on a PCIEx1 to PCI adapter. I only use the external 50-pin SCSI interface. That can only take 1.1GB MO disks though, and i only have about 30 of them. they are everlasting though, its the drives that dies first lol anything of extreme inportance goes to that.
    ( i have a backup drive still in its anti-stat bag).
    Next step would be LTO storage, as with LTFS you can now use them in standard Windows 10. 50€ for 1.5TB of long term storage.
    2000€ for the drive though...

    Still, have a look at XPenology, and maybe even give it a try ? only need to boot to usb to check if it works on your hardware. Thats what i did, and then just went from there.
    https://xpenology.com/forum/
    one restriction, it would seem the last XPenology compatible version of the Synology DSM software is 6.1.x
    Last edited by Evildead666; 8th July 2018 at 12:29.
    PC-1 Fractal Design Arc Mini R2, i5-3570K@4.2GHz, ASRock Extreme4-M, 4x4Gb GSkill DDR3-2133, 850Pro 256Gb SSD, 840EVO 250Gb, 2xWD Black 750Gb HDD (Raid-0), GigaMO, AMD Vega64, Seasonic 850W Gold, H100i for CPU + Black Ice SR-1/Laing DDC/EKWB 240 Loop for Vega64.
    Nas : i3/itx/4x4TB Raid6
    +++ : MGE UPS 650VA (Nas+switch) + 1600VA (PC-1+cooling)

  4. #94
    Crabby Smurf Umfriend's Avatar
    Join Date
    Mar 2001
    Location
    Netherlands
    Posts
    6,864

    Default

    Quote Originally Posted by VJ View Post
    Reviving this thread as my stay in Spain is nearing its end (just half a year to go... amazing how time flies) and I'll soon be back in Warsaw taking care of organizing my data properly.


    Umfriend: I have some questions regarding Drive Pool. Is it possible to do something like this:

    2 x 2TB harddisk (let's call them 2a and 2b), 1x 4TB harddisk (let's call it 4), 1x 8 TB harddisk (let's call it 8)

    Duplicate a folder on 2a + 4 + 8 and another folder on 2a + 8 (I would do something similar with 2b+4+8, 2b+8 and 4+8). If the 8 TB disk is slower does it affect the overal write speed? If so, is there a way of making it duplicate in the background? I think the plugin "SSD optimizer" seems to allow for that, or maybe the "Ordered file placement". Would this work?

    Can you set a network mount as a harddisk, or does this need CloudDrive? What happens when it is not available? (thinking of using my old computer to hold some duplicates, but it may be just easier to use some synchronization rather than DrivePool).

    Thanks!
    Missed this! No, a network mount can not be part of a Pool AFAIK. I don;t understand the 2a + 4 + 8 and 2a + 8 bit. How many duplicates do you want and am I right that you have 4HDDs in total? Yes, if the 8TB HDD is slow than that will govern write speeds. SSD Optimizer may help though, that is what someone does to feed from faster HDDs to 8TB Seagate Archive drives, which can be slow (if you write more than 20GB in a short time).

    I am not familiar with CloudDrive. It may help using network mounts but I think you'd need some sort of API like the cloud storage providers have. Best to ask at their forums.
    Join MURCs Distributed Computing effort for Rosetta@Home and help fight Alzheimers, Cancer, Mad Cow disease and rising oil prices.
    [...]the pervading principle and abiding test of good breeding is the requirement of a substantial and patent waste of time. - Veblen

  5. #95
    Moderator VJ's Avatar
    Join Date
    Mar 2001
    Location
    Belgium/Poland/Spain
    Posts
    9,392

    Default

    Thanks!
    When I return to Poland, the plan is to fully make order in our computers and data storage (data are currently spread over 2 computers with in total 6 internal disks and over 6 external disks). One of the computers is the old dual Xeon, which I would not like to have permanently on as storage server (noise + power).

    I know that at the moment a lot of data is duplicated, but lacking a proper strategy and automation, it is quite a mess... We mainly have four types of data: own made photos/videos (for which we want proper redundancy and backup), work related files (articles, source code; which I also don't want to lose), various support files (software/manuals, usually can be downloaded again but I like to have my own offline copy - has saved me a couple of times) and then our music/video collection (for which redundancy is less important although I'd hate to have to re-rip it all over again). I'm currently examining the data flows we have (e.g. the difference with how photos/videos from cameras and mobiles are treated).

    I am thinking of re-purposing one of the external disks as a storage connected to our router as sort of a private cloud; its main purpose would be to serve as universal entry point for data that comes from mobile devices (e.g. photos from phones or laptops): the main computer could do a one-way sync with it in order to integrate the data. The reason for using the router as a private cloud is that there is mobile phone software that integrates with it easily. It would also be an easy point to upload data using e.g. a laptop. It would not be the fastest entry point though, but for mobile data it should suffice. Other data (e.g. from my camera) would go directly to the main computer. Redundancy depends on the type of data. And then at regular times this could be synchronized with the other computer (as a type of backup).
    pixar
    Dream as if you'll live forever. Live as if you'll die tomorrow. (James Dean)

  6. #96
    Super MURCer UtwigMU's Avatar
    Join Date
    Jul 2002
    Location
    Sin City
    Posts
    4,960

    Default

    I've built ZFS NAS.
    Fractal Define Mini + additional Fractal fan (2 intake, 1 exhaust)
    Corsair RM650x modular gold, silent mode on low load
    Supermicro X9SCM-F - onboard Matrox G200
    32GB ECC
    Pentium G
    Noctua NH-U12S
    IBM (LSI) 8 port SAS flashed to LSI and IT mode - flashing this in UEFI shell was a bit of work
    4x8TB WD RED
    2x Samsung 860 (tried Sandisk SSDPlus at first 120GB but boot volume would become corrupt - read that this drive has problems with FreeBSD)

    While web interface is not streamlined everywhere (you need to create dataset and set permissions there and share on top of it where you can't set permissions) and BSD is a bit quirky (sometimes it would not get IP), it's nice overall. I tested simulated drive failure and replacement, volume import, config restore, boot RAID, email alerts.

    Resilver (equivalent of RAID rebuild) with about 1TB of data on volume completed in about half an hour which is much better than traditional RAID where with this drive size it would take 8-20h. You can also schedule scrubs (read all data, check for errors and correct from parity). I would say ZFS is for you if you know Unix/Linux CLI and have basic understanding of Unix/Linux administration. While you don't need it to set up, you will probably need it if something goes wrong. It's also good to know what GUI is abstracting from you and what goes on in the background (Unix permissions, users, groups).

    I'm tempted to add 2 more drives in the future (will need to back up and rebuild) for RAIDZ2 (RAID6 equivalent), while I'll exceed 1GB RAM / 1TB storage, I'm not running jails and don't have multiple users or VMs.

    You can upgrade where you replace all drives with larger ones (you can leave old drive in until end of rebuild if you have enough SAS/SATA ports). Once all drives have been replaced, extra space becomes visible.

    Windows shares and setting share permissions from Windows Explorer (provided you're share owner) works perfectly, it's using SMBv3. I hit 100MB/s over single Gb NIC with large files and 30MB/s with MP3s.

    If you have a spare box or hypervisor with 8GB free you can try FreeNAS (not for production - just for testing, for production you want bare metal and ECC) - the system install from USB takes under 3 minutes.

    I'm copying data from my old 2k3 server which has been sitting powered off in the corner. Then I will copy data from laptops and external drives, SD Cards and sort it.

    I set shares:
    backup where I have RW - here I will dump my own created files from my laptops
    archive where I have read only (created another user with RW for storing data) - here I will move finished projects and photos
    media (movies, music, pdfs, isos) - here I also have read only

    I plan to buy couple 5TB external drives. One for media and other for archive + backup. I'll take these off site (like I do with my esx hosting mail server - 2 drives, daily backup, rotate drives about once a month)
    Last edited by UtwigMU; Yesterday at 07:33.

Thread Information

Users Browsing this Thread

There are currently 2 users browsing this thread. (0 members and 2 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •