Search This Blog

Wednesday, October 28, 2015

Day 2 - VeeamOn 2015

Today marked the 2nd day of VeeamOn 2015 events. The following was our short note taking about every session that we've attended. (Sorry for using point form & demo video quality may not be cleared as we recorded it using smartphone.)

Session 1:- Veeam Backup at Full Throttle (Luca, Tom)

How to optimize VBR in term of design and implementation?

Default settings - it is good but to increase performance it is better to fine tune

Example: car. You need to fine tune to give you better performance.

 What you can tune?

  • VBS
  • Proxy
  • Backup repository
  • Physical & Hypervisor

 vSphere : Look for

  • Vcenter VCSA- appliance size & performance
  • Windows version - SQL (Express- small( 1 core/1 vcpu)- not multicore, Local, Remote)

 Check -> number of concurrent connection created by Veeam

Vcenter has connection limit - use vcenter client (Backup connection establish from Backup Job- min 2.)

Per default , maximum is 500 connection. Can increase the limit but will incur additional memory usage

 SQL connection - SQL Express ( 1 socket but multicore)

 ESXi limit on NBD. NFC is limited on resources, especially on 1GB network. Recommend use 10G Network (Use esxi 5.0 update 2)

-increase number of buffer from 2048 to 4096

Reduce buffer cache flush interval from 30 seconds to 20 seconds

Veeam fine tuning

Default value for small environment

Large env - fine tune!

Snapshot per datastore. Limit max amount of active VM snapshot per datastore to prevent it from being overfilled with snapshot delta. More concurrent snapshot, will incur additional space consumption

Default value is 4.  Change it with MaxSnapshotPerDataStore (REG_DWORD)

-Available since build 7.0.0771 (V7 Patch 2)
Snapshot per vCenter - limit max amount of active VM snashot per vcenter when using BfSS

Default value = 8 (in v7), or 20(in v8)

-Available since build 7.0.0771 (V7 Patch 2)

Snapshot per ESXi - limit max amount of active VM snashot per ESXi

 Remove the snapshot

If you have fast storage, you can increase the limit

MaxConcurrentDeletingSnapshotsForCluster (DWORD) - default 4

MaxConcurrentDeletingSnapshotsForHost (DWORD)- default 2

 Backup from Storage Snapshot

  • In v9, can configure limit on the snapshot of the VM.
  • Direct NFS

 Optimizing proxy & repository

  • Jumbo frames
  • Use VMXNet3 for virtual proxies but not PVSSCSI
  • Tune NICs for throughput
  • Look out for hardware optimization that introduce latency

 Proxy sizing
  • 2GB per core is ideal
  • Estimated throughput is 50MB/s for each core
  • If individual backup seem slow: vddkPreReadBufferSize - default 4MB. Increase can improve full performance (8MB or 16 MB)

Proxy recommendation

  • Rule of thumb - 30-50 VM per core
  • 4 vcpu / 8GB is great
  • Increase proxy when require

 Repository Sizing

  • 4GB ram per core for large jobs
  • Fast storage connectivity (local attach or 10GBE)
  • Single server can be configured for multiple Veeam repository

Which type to choose?

  • Local disk / Nas/Dedupe
  • Physical server + Fast Locak Disk (windows /linux)
  • Fast caching RAID is the best, large stripe size (128K or greater) and SSD caching

 Dedupe (great for long term retention)
  • support for DDBoast
  • Support for Catalyst on HPstoreOnce
  • Support ExaGrid

V9 - Check out : New per VM Backup Chain features.

 Session 2:- Core Platform Availability in Windows Server 2016 (Ben Armstrong)


Session 3:- Veeam General Session

New Product announcement :- Veeam Backup for Linux. - allow you to protect and recover Linux workload similar like Veeam Endpoint Backup. Since Linux do not has VSS, Veeam using it own built-in change tracking. Supported workload is Debian / Redhat. Currently it is under beta and it's FREE!. If you're interested to test it out, please register from here:-
Customer also can leverage on existing Veeam Repositories as backup target.
Session 4:Backup Repository Best Practices 2015 Edition (Anton)

Don't understand the performance. Common issue - slow backup performance

5 Factors:

  • Reliability
  • Fast backup
  • Fast restores
  • DR from complete storage lost
  • Lowest cost

Cost control

  • Limit number of restore point on primary storage
  • Using lower cost (per TB) secondary storage for long term retention
  • At least 1 copy of backup
  • Selective create offsite copies on tape in DR or in cloud

Backup repository - Low end

  • Any windows or linux server
  • Physical server - local storage, DAS (JBOD) , SAN LUN
Virtual Server storage
  • Iscsi LUN via in guest iSCSI
  • pDRM disk (require vsphere 5)
Backup repository - High end

Deduplication storage

  • EMC data domain
  • Exagrid
  • HP StoreOnce

Top above brand, but Veeam will work with any

VTL will be better (faster) option for backup

Backup Repository to Avoid

  • Low end NAS and appliance (reliability)
    • If stuck, use ISCSI instead of file protocol
  • SMB (CIFS) network shares on Windows Server  (reliability & performance)
  • VMDK on VMFS (recoverability)
    • Dependent on vSphere
  • Windows Server 2012 Deduplication (scalability)

 Immediate Future Support

  • Windows Server 2016 deduplication
  • REFS 2.0 - built on data corruption guard

 RAW Disk (Internal, DAS, SAN Storage)

  • Raid 10 whenever you can  (2x write penalty, but capacity suffer)
  • RAID 5 - next economical choice (4x write penalty but greater risk
  • RAID 6 - has most severe performance overhead( 6x write penalty)

Maximum performance per spindle (lowest 7.2k, highest -15k SAS)

 RAID volume

  • RAID stripe size - typical i/o for veeam is 256KB to 512KB
  • Windows Server 2012 default to 64KB. That's a lot of wasted I/O
  • At least 128KB stripe size is highly recommended.

 RAID array

  • Fill as many drives as possible from the start to avoid expansion
  • Low end storage has significant performance degradation when expanding.

File System:

  • NTFS - recommend 64KB block size
  • 16TB max file size limit before Win 2012 (which increase it to 256 TB)
  • Format with /L to enable larger file record (enable large backup files on very large volumes)
  • Refs 2.0

 Backup Job Settings

  • Always a performance vs disk space trade off
  • Reverse incremental backup mode is 3x I/ per block
  • Consider forever forward incremental backup mode instead
  • Evaluate transform performance

 Repository Load

  • Limit concurrent jobs to a reasonable amount
  • -use ingest rate throttling for cross SAN backup

 Deduplication Storage


  • True global dedupe
  • Lowest cost per TB


  • Random access performance
  • Poor ingest rates (bad as primary target)
  • Require significant upfront investment

 Best practice as primary storage

  • Do not use deduplication storage as primary backup repository

But if you have to

  • Leverage on vendor specific integration
  • Use backup modes without full backup transformation)
  • Use active full instead synthetic full (if possible)
  • If backup performance is still bad, consider VTL
  • Use 16TB backup storage optimization for 4MB blocks
  • Parallel processing may impact dedupe ration
    • Limit to 1 task, instead of disabling

 Best practice as secondary storage

  • Leverage on vendor specific integration
  • No integration with your storage choice?
  • Test backup copy retention processing performance. If too slow, consider using ACTIVE FUL OF BACKUP Copy Jobs
  • Too late, already invested?
    • Use as primary storage and leverage native replication

Best practice for Backup Job settings

  • Built in deduplication
    • Keep on for best performance (except lowest end devices)
  • Compression
    • Instead of disable , keep optimal compression enabled in job and use decompress before storing repository action
    • Dedupe friendly compression is not so friendly anymore

Why I love Tape?

  • Cheaper
  • Reliable
  • Read only
  • Customer success !

Tape is not dead!

What can go wrong with disk?

-  Storage level corruption

  • RAID controller ae your worst enemies
  • Firmware and software bug are common


-Unhappy Insider - unhappy staff

 Part of 3-2-1 Rule

-3 copies - 2 different media ,1 of them is offsite

 -2 mean completely different storage types - disk & tape

 Storage Based Replication

Cons - replicate bad data just as good as it replicated

Your backup remains in a single fault domain (3-2-1 is not meet)

 Backup Copy vs Storage Based copy


  • Break the data loop
  • Validates all source data
  • Include backup file health check with auto healing
  • Efficient backup optimization

How to make tape out of disk?

Low end:-

  • Use rotated drives

Mid range-

  • Keep an offsite copy off-prem (cloud)

 High end:

  • Use hardware based WORM solutions

 Check out v9 :- Scale Out backup Repository - v9 (virtualize backup repository)

That's all for today. Stay Tuned for Day 3